2021-08-21T14:51:48.635 INFO:root:teuthology version: 1.1.0-321319b1 2021-08-21T14:51:48.636 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2021-08-21_14:21:15-rados-wip-yuri-master-8.20.21-distro-basic-smithi/6350476 --verbose --name yuriw-2021-08-21_14:21:15-rados-wip-yuri-master-8.20.21-distro-basic-smithi --description rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2021-08-21_14:21:15-rados-wip-yuri-master-8.20.21-distro-basic-smithi/6350476/orig.config.yaml 2021-08-21T14:51:48.673 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-08-21T14:51:48.722 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-08-21_14:21:15-rados-wip-yuri-master-8.20.21-distro-basic-smithi/6350476 branch: wip-yuri-master-8.20.21 description: rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} email: ceph-qa@ceph.io first_in_suite: false job_id: '6350476' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2021-08-21_14:21:15-rados-wip-yuri-master-8.20.21-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-yuri-master-8.20.21 ceph: conf: global: mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: cae1ffd55a16af4627c36508fde05bf24312fad6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: cae1ffd55a16af4627c36508fde05bf24312fad6 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri-master-8.20.21 sha1: cae1ffd55a16af4627c36508fde05bf24312fad6 owner: scheduled_yuriw@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 sha1: cae1ffd55a16af4627c36508fde05bf24312fad6 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri-master-8.20.21 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_cae1ffd55a16af4627c36508fde05bf24312fad6/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: cae1ffd55a16af4627c36508fde05bf24312fad6 targets: smithi165.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNjqTYpGqmRw5Cy5aurn0f7qY/DYXAUcAWc1XqSqr2zsn3Omu0698Q7pGEs/T4d2ZfSm5koCzjJtYm+DQGdOhPM2cpNichQvhJaE0BCaW5Y5TJcbNDdWp+WOH/9yZOe4NJnS7Jh+I75j6hJemGJIZRd71a7mumxgaKbhtfmseVpcdOzX3YhVBwnzwS/SQaeO8YIlON7lMwv1EwNI7l0F9PVx4F+f9OBZsHamWlKTvs/daugwOmeOSSpcoQZQPZ9Zw8Wmh5T+YfZtqyW/EDhuBg01t2o1BSq0gRRuWpBenT7ax2qKw99xHU+QXpiCk6oM7lbqvZ9ANLgOq0MB5yLxA7PG8EhTnq81QzQcpudqeUl25wdIWe7BWb5lHdKG4dlqm8BoKPMSOW8fUd7C6ZDNab9vORjwqPW2ra9aX7894l5S8a/GiAx1gwBStBF6LntI+yk01kmMMUjljwV0K0DK9yOYbEoWstOmgniJNSk8JmqxpWFSwkmROuMIpqlfDKcp0= tasks: - install: null - ceph: conf: osd: osd min pg log entries: 5 osd_fast_fail_on_connection_refused: false log-ignorelist: - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - osd_recovery.test_incomplete_pgs: null teuthology_branch: master teuthology_sha1: 321319b12ea4ff9b63c7655015a3156de2c3b279 timestamp: 2021-08-21_14:21:15 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.30581 2021-08-21T14:51:48.723 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_cae1ffd55a16af4627c36508fde05bf24312fad6/qa; will attempt to use it 2021-08-21T14:51:48.723 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_cae1ffd55a16af4627c36508fde05bf24312fad6/qa/tasks 2021-08-21T14:51:48.724 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-08-21T14:51:48.726 INFO:teuthology.task.internal:Checking packages... 2021-08-21T14:51:48.746 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'cae1ffd55a16af4627c36508fde05bf24312fad6' 2021-08-21T14:51:48.746 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-08-21T14:51:48.747 INFO:teuthology.packaging:ref: None 2021-08-21T14:51:48.747 INFO:teuthology.packaging:tag: None 2021-08-21T14:51:48.748 INFO:teuthology.packaging:branch: wip-yuri-master-8.20.21 2021-08-21T14:51:48.748 INFO:teuthology.packaging:sha1: cae1ffd55a16af4627c36508fde05bf24312fad6 2021-08-21T14:51:48.748 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-yuri-master-8.20.21 2021-08-21T14:51:48.896 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-7135-gcae1ffd5-1focal 2021-08-21T14:51:48.897 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-08-21T14:51:48.910 INFO:teuthology.task.internal:no buildpackages task found 2021-08-21T14:51:48.911 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-08-21T14:51:48.954 INFO:teuthology.task.internal:Saving configuration 2021-08-21T14:51:48.966 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-08-21T14:51:48.982 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-21T14:51:48.999 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-08-21 14:46:17.604696', 'description': '/home/teuthworker/archive/yuriw-2021-08-21_14:21:15-rados-wip-yuri-master-8.20.21-distro-basic-smithi/6350476', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOzfN8LZgQZj6WUZLV36XTnfBhc9ZxPkqa5VFUiGuVXD', 'os_version': '20.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi165.front.sepia.ceph.com'} 2021-08-21T14:51:49.000 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-08-21T14:51:49.007 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2021-08-21T14:51:49.007 INFO:teuthology.run_tasks:Running task console_log... 2021-08-21T14:51:49.053 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5637b90b70>, signals=[15]) 2021-08-21T14:51:49.055 INFO:teuthology.run_tasks:Running task internal.connect... 2021-08-21T14:51:49.062 INFO:teuthology.task.internal:Opening connections... 2021-08-21T14:51:49.062 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2021-08-21T14:51:49.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:51:49.112 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-08-21T14:51:49.122 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2021-08-21T14:51:49.237 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2021-08-21T14:51:49.238 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2021-08-21T14:51:49.243 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="Ubuntu" 2021-08-21T14:51:49.243 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="20.04.2 LTS (Focal Fossa)" 2021-08-21T14:51:49.243 INFO:teuthology.orchestra.run.smithi165.stdout:ID=ubuntu 2021-08-21T14:51:49.244 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE=debian 2021-08-21T14:51:49.244 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS" 2021-08-21T14:51:49.244 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="20.04" 2021-08-21T14:51:49.244 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://www.ubuntu.com/" 2021-08-21T14:51:49.245 INFO:teuthology.orchestra.run.smithi165.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-08-21T14:51:49.245 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-08-21T14:51:49.245 INFO:teuthology.orchestra.run.smithi165.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-08-21T14:51:49.245 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_CODENAME=focal 2021-08-21T14:51:49.246 INFO:teuthology.orchestra.run.smithi165.stdout:UBUNTU_CODENAME=focal 2021-08-21T14:51:49.246 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2021-08-21T14:51:49.261 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-08-21T14:51:49.273 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-08-21T14:51:49.300 INFO:teuthology.task.internal:Checking for old test directory... 2021-08-21T14:51:49.300 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2021-08-21T14:51:49.306 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-08-21T14:51:49.325 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-08-21T14:51:49.325 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2021-08-21T14:51:49.355 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-08-21T14:51:49.384 INFO:teuthology.run_tasks:Running task kernel... 2021-08-21T14:51:49.402 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}} 2021-08-21T14:51:49.402 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-08-21T14:51:49.403 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2021-08-21T14:51:49.409 INFO:teuthology.orchestra.run.smithi165.stdout:5.4.0-66-generic 2021-08-21T14:51:49.410 INFO:teuthology.task.kernel:Running kernel on smithi165: 5.4.0-66-generic 2021-08-21T14:51:49.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get clean 2021-08-21T14:51:49.576 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get update 2021-08-21T14:51:49.738 INFO:teuthology.orchestra.run.smithi165.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2021-08-21T14:51:49.850 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-08-21T14:51:49.933 INFO:teuthology.orchestra.run.smithi165.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2021-08-21T14:51:50.028 INFO:teuthology.orchestra.run.smithi165.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [274 kB] 2021-08-21T14:51:50.083 INFO:teuthology.orchestra.run.smithi165.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [822 kB] 2021-08-21T14:51:50.109 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [161 kB] 2021-08-21T14:51:50.112 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [8,440 B] 2021-08-21T14:51:50.112 INFO:teuthology.orchestra.run.smithi165.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [18.0 kB] 2021-08-21T14:51:50.113 INFO:teuthology.orchestra.run.smithi165.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [364 kB] 2021-08-21T14:51:50.119 INFO:teuthology.orchestra.run.smithi165.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [52.1 kB] 2021-08-21T14:51:50.120 INFO:teuthology.orchestra.run.smithi165.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [484 B] 2021-08-21T14:51:50.121 INFO:teuthology.orchestra.run.smithi165.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [640 kB] 2021-08-21T14:51:50.129 INFO:teuthology.orchestra.run.smithi165.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [509 kB] 2021-08-21T14:51:50.135 INFO:teuthology.orchestra.run.smithi165.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [100 kB] 2021-08-21T14:51:50.136 INFO:teuthology.orchestra.run.smithi165.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [12.3 kB] 2021-08-21T14:51:50.270 INFO:teuthology.orchestra.run.smithi165.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB] 2021-08-21T14:51:50.395 INFO:teuthology.orchestra.run.smithi165.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [522 kB] 2021-08-21T14:51:50.628 INFO:teuthology.orchestra.run.smithi165.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,170 kB] 2021-08-21T14:51:50.765 INFO:teuthology.orchestra.run.smithi165.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [252 kB] 2021-08-21T14:51:50.786 INFO:teuthology.orchestra.run.smithi165.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [13.9 kB] 2021-08-21T14:51:50.786 INFO:teuthology.orchestra.run.smithi165.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [410 kB] 2021-08-21T14:51:50.805 INFO:teuthology.orchestra.run.smithi165.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [19.3 kB] 2021-08-21T14:51:50.805 INFO:teuthology.orchestra.run.smithi165.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [58.7 kB] 2021-08-21T14:51:50.809 INFO:teuthology.orchestra.run.smithi165.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [480 B] 2021-08-21T14:51:50.809 INFO:teuthology.orchestra.run.smithi165.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [851 kB] 2021-08-21T14:51:50.850 INFO:teuthology.orchestra.run.smithi165.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [632 kB] 2021-08-21T14:51:50.877 INFO:teuthology.orchestra.run.smithi165.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [179 kB] 2021-08-21T14:51:50.882 INFO:teuthology.orchestra.run.smithi165.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [18.6 kB] 2021-08-21T14:51:50.931 INFO:teuthology.orchestra.run.smithi165.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,228 B] 2021-08-21T14:51:50.931 INFO:teuthology.orchestra.run.smithi165.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.0 kB] 2021-08-21T14:51:50.932 INFO:teuthology.orchestra.run.smithi165.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [6,888 B] 2021-08-21T14:51:50.932 INFO:teuthology.orchestra.run.smithi165.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [652 B] 2021-08-21T14:51:54.167 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 7,456 kB in 2s (4,900 kB/s) 2021-08-21T14:51:55.317 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2021-08-21T14:51:55.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-08-21T14:51:55.408 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2021-08-21T14:51:55.555 INFO:teuthology.orchestra.run.smithi165.stdout:Building dependency tree... 2021-08-21T14:51:55.556 INFO:teuthology.orchestra.run.smithi165.stdout:Reading state information... 2021-08-21T14:51:55.574 INFO:teuthology.orchestra.run.smithi165.stderr:E: Unable to locate package linux-image-current-generic 2021-08-21T14:51:55.576 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-08-21T14:51:55.577 DEBUG:teuthology.orchestra.run.smithi165:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-08-21T14:51:55.645 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2021-08-21T14:51:55.808 INFO:teuthology.orchestra.run.smithi165.stdout:Building dependency tree... 2021-08-21T14:51:55.808 INFO:teuthology.orchestra.run.smithi165.stdout:Reading state information... 2021-08-21T14:51:55.949 INFO:teuthology.orchestra.run.smithi165.stdout:The following additional packages will be installed: 2021-08-21T14:51:55.950 INFO:teuthology.orchestra.run.smithi165.stdout: linux-generic linux-headers-5.4.0-81 linux-headers-5.4.0-81-generic 2021-08-21T14:51:55.950 INFO:teuthology.orchestra.run.smithi165.stdout: linux-headers-generic linux-image-5.4.0-81-generic 2021-08-21T14:51:55.950 INFO:teuthology.orchestra.run.smithi165.stdout: linux-modules-5.4.0-81-generic linux-modules-extra-5.4.0-81-generic 2021-08-21T14:51:55.951 INFO:teuthology.orchestra.run.smithi165.stdout:Suggested packages: 2021-08-21T14:51:55.951 INFO:teuthology.orchestra.run.smithi165.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2021-08-21T14:51:55.951 INFO:teuthology.orchestra.run.smithi165.stdout:Recommended packages: 2021-08-21T14:51:55.952 INFO:teuthology.orchestra.run.smithi165.stdout: thermald 2021-08-21T14:51:55.979 INFO:teuthology.orchestra.run.smithi165.stdout:The following NEW packages will be installed: 2021-08-21T14:51:55.979 INFO:teuthology.orchestra.run.smithi165.stdout: linux-headers-5.4.0-81 linux-headers-5.4.0-81-generic 2021-08-21T14:51:55.979 INFO:teuthology.orchestra.run.smithi165.stdout: linux-image-5.4.0-81-generic linux-modules-5.4.0-81-generic 2021-08-21T14:51:55.980 INFO:teuthology.orchestra.run.smithi165.stdout: linux-modules-extra-5.4.0-81-generic 2021-08-21T14:51:55.980 INFO:teuthology.orchestra.run.smithi165.stdout:The following packages will be upgraded: 2021-08-21T14:51:55.981 INFO:teuthology.orchestra.run.smithi165.stdout: linux-generic linux-headers-generic linux-image-generic 2021-08-21T14:51:56.161 INFO:teuthology.orchestra.run.smithi165.stdout:3 upgraded, 5 newly installed, 0 to remove and 168 not upgraded. 2021-08-21T14:51:56.161 INFO:teuthology.orchestra.run.smithi165.stdout:Need to get 75.6 MB of archives. 2021-08-21T14:51:56.161 INFO:teuthology.orchestra.run.smithi165.stdout:After this operation, 378 MB of additional disk space will be used. 2021-08-21T14:51:56.162 INFO:teuthology.orchestra.run.smithi165.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-81-generic amd64 5.4.0-81.91 [14.8 MB] 2021-08-21T14:51:57.243 INFO:teuthology.orchestra.run.smithi165.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-81-generic amd64 5.4.0-81.91 [8,986 kB] 2021-08-21T14:51:57.500 INFO:teuthology.orchestra.run.smithi165.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-81-generic amd64 5.4.0-81.91 [39.4 MB] 2021-08-21T14:51:58.797 INFO:teuthology.orchestra.run.smithi165.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.81.85 [1,900 B] 2021-08-21T14:51:58.798 INFO:teuthology.orchestra.run.smithi165.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.81.85 [2,572 B] 2021-08-21T14:51:58.798 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-81 all 5.4.0-81.91 [11.0 MB] 2021-08-21T14:51:59.140 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-81-generic amd64 5.4.0-81.91 [1,410 kB] 2021-08-21T14:51:59.165 INFO:teuthology.orchestra.run.smithi165.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.81.85 [2,440 B] 2021-08-21T14:51:59.584 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 75.6 MB in 3s (23.8 MB/s) 2021-08-21T14:51:59.745 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-modules-5.4.0-81-generic. 2021-08-21T14:52:01.230 INFO:teuthology.orchestra.run.smithi165.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101688 files and directories currently installed.) 2021-08-21T14:52:01.234 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../0-linux-modules-5.4.0-81-generic_5.4.0-81.91_amd64.deb ... 2021-08-21T14:52:01.334 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-modules-5.4.0-81-generic (5.4.0-81.91) ... 2021-08-21T14:52:03.423 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-image-5.4.0-81-generic. 2021-08-21T14:52:03.436 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../1-linux-image-5.4.0-81-generic_5.4.0-81.91_amd64.deb ... 2021-08-21T14:52:03.531 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-image-5.4.0-81-generic (5.4.0-81.91) ... 2021-08-21T14:52:04.573 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-81-generic. 2021-08-21T14:52:04.588 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-81-generic_5.4.0-81.91_amd64.deb ... 2021-08-21T14:52:04.626 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-modules-extra-5.4.0-81-generic (5.4.0-81.91) ... 2021-08-21T14:52:09.759 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../3-linux-generic_5.4.0.81.85_amd64.deb ... 2021-08-21T14:52:10.252 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-generic (5.4.0.81.85) over (5.4.0.66.69) ... 2021-08-21T14:52:10.596 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.81.85_amd64.deb ... 2021-08-21T14:52:10.745 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-image-generic (5.4.0.81.85) over (5.4.0.66.69) ... 2021-08-21T14:52:11.080 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-headers-5.4.0-81. 2021-08-21T14:52:11.094 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../5-linux-headers-5.4.0-81_5.4.0-81.91_all.deb ... 2021-08-21T14:52:11.157 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-headers-5.4.0-81 (5.4.0-81.91) ... 2021-08-21T14:52:17.392 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package linux-headers-5.4.0-81-generic. 2021-08-21T14:52:17.415 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../6-linux-headers-5.4.0-81-generic_5.4.0-81.91_amd64.deb ... 2021-08-21T14:52:17.458 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-headers-5.4.0-81-generic (5.4.0-81.91) ... 2021-08-21T14:52:20.053 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.81.85_amd64.deb ... 2021-08-21T14:52:20.181 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking linux-headers-generic (5.4.0.81.85) over (5.4.0.66.69) ... 2021-08-21T14:52:20.915 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-modules-5.4.0-81-generic (5.4.0-81.91) ... 2021-08-21T14:52:22.086 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-image-5.4.0-81-generic (5.4.0-81.91) ... 2021-08-21T14:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-81-generic 2021-08-21T14:52:23.206 INFO:teuthology.orchestra.run.smithi165.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-81-generic 2021-08-21T14:52:23.346 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-headers-5.4.0-81 (5.4.0-81.91) ... 2021-08-21T14:52:23.497 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-modules-extra-5.4.0-81-generic (5.4.0-81.91) ... 2021-08-21T14:52:24.673 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-image-generic (5.4.0.81.85) ... 2021-08-21T14:52:24.808 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-headers-5.4.0-81-generic (5.4.0-81.91) ... 2021-08-21T14:52:24.959 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-headers-generic (5.4.0.81.85) ... 2021-08-21T14:52:25.101 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up linux-generic (5.4.0.81.85) ... 2021-08-21T14:52:25.228 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for linux-image-5.4.0-81-generic (5.4.0-81.91) ... 2021-08-21T14:52:25.403 INFO:teuthology.orchestra.run.smithi165.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-08-21T14:52:25.403 INFO:teuthology.orchestra.run.smithi165.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-81-generic 2021-08-21T14:52:46.977 INFO:teuthology.orchestra.run.smithi165.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-08-21T14:52:46.978 INFO:teuthology.orchestra.run.smithi165.stdout:Sourcing file `/etc/default/grub' 2021-08-21T14:52:47.018 INFO:teuthology.orchestra.run.smithi165.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2021-08-21T14:52:47.035 INFO:teuthology.orchestra.run.smithi165.stdout:Generating grub configuration file ... 2021-08-21T14:52:47.393 INFO:teuthology.orchestra.run.smithi165.stdout:Found linux image: /boot/vmlinuz-5.4.0-81-generic 2021-08-21T14:52:47.425 INFO:teuthology.orchestra.run.smithi165.stdout:Found initrd image: /boot/initrd.img-5.4.0-81-generic 2021-08-21T14:52:47.842 INFO:teuthology.orchestra.run.smithi165.stdout:Found linux image: /boot/vmlinuz-5.4.0-66-generic 2021-08-21T14:52:47.854 INFO:teuthology.orchestra.run.smithi165.stdout:Found initrd image: /boot/initrd.img-5.4.0-66-generic 2021-08-21T14:52:50.285 INFO:teuthology.orchestra.run.smithi165.stdout:done 2021-08-21T14:52:50.685 DEBUG:teuthology.orchestra.run.smithi165:> dpkg -s linux-image-generic 2021-08-21T14:52:50.711 INFO:teuthology.orchestra.run.smithi165.stdout:Package: linux-image-generic 2021-08-21T14:52:50.711 INFO:teuthology.orchestra.run.smithi165.stdout:Status: install ok installed 2021-08-21T14:52:50.711 INFO:teuthology.orchestra.run.smithi165.stdout:Priority: optional 2021-08-21T14:52:50.711 INFO:teuthology.orchestra.run.smithi165.stdout:Section: kernel 2021-08-21T14:52:50.712 INFO:teuthology.orchestra.run.smithi165.stdout:Installed-Size: 18 2021-08-21T14:52:50.712 INFO:teuthology.orchestra.run.smithi165.stdout:Maintainer: Ubuntu Kernel Team 2021-08-21T14:52:50.712 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: amd64 2021-08-21T14:52:50.712 INFO:teuthology.orchestra.run.smithi165.stdout:Source: linux-meta 2021-08-21T14:52:50.712 INFO:teuthology.orchestra.run.smithi165.stdout:Version: 5.4.0.81.85 2021-08-21T14:52:50.713 INFO:teuthology.orchestra.run.smithi165.stdout:Provides: virtualbox-guest-modules (= 6.1.16-dfsg-6~ubuntu1.20.04.2), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.10) 2021-08-21T14:52:50.713 INFO:teuthology.orchestra.run.smithi165.stdout:Depends: linux-image-5.4.0-81-generic, linux-modules-extra-5.4.0-81-generic, linux-firmware, intel-microcode, amd64-microcode 2021-08-21T14:52:50.713 INFO:teuthology.orchestra.run.smithi165.stdout:Recommends: thermald 2021-08-21T14:52:50.713 INFO:teuthology.orchestra.run.smithi165.stdout:Description: Generic Linux kernel image 2021-08-21T14:52:50.713 INFO:teuthology.orchestra.run.smithi165.stdout: This package will always depend on the latest generic kernel image 2021-08-21T14:52:50.714 INFO:teuthology.orchestra.run.smithi165.stdout: available. 2021-08-21T14:52:50.714 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-66-generic Expected: 5.4.0-81-generic 2021-08-21T14:52:50.714 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-08-21T14:52:50.715 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-08-21T14:52:50.715 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2021-08-21T14:52:50.715 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi165.front.sepia.ceph.com, path=None, version=distro) 2021-08-21T14:52:50.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get clean 2021-08-21T14:52:50.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get update 2021-08-21T14:52:50.969 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-08-21T14:52:51.076 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2021-08-21T14:52:51.159 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2021-08-21T14:52:51.241 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2021-08-21T14:52:52.418 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2021-08-21T14:52:52.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-08-21T14:52:52.507 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2021-08-21T14:52:52.651 INFO:teuthology.orchestra.run.smithi165.stdout:Building dependency tree... 2021-08-21T14:52:52.652 INFO:teuthology.orchestra.run.smithi165.stdout:Reading state information... 2021-08-21T14:52:52.671 INFO:teuthology.orchestra.run.smithi165.stderr:E: Unable to locate package linux-image-current-generic 2021-08-21T14:52:52.672 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-08-21T14:52:52.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-08-21T14:52:52.736 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2021-08-21T14:52:52.897 INFO:teuthology.orchestra.run.smithi165.stdout:Building dependency tree... 2021-08-21T14:52:52.897 INFO:teuthology.orchestra.run.smithi165.stdout:Reading state information... 2021-08-21T14:52:53.057 INFO:teuthology.orchestra.run.smithi165.stdout:linux-image-generic is already the newest version (5.4.0.81.85). 2021-08-21T14:52:53.058 INFO:teuthology.orchestra.run.smithi165.stdout:0 upgraded, 0 newly installed, 0 to remove and 168 not upgraded. 2021-08-21T14:52:53.059 DEBUG:teuthology.orchestra.run.smithi165:> dpkg -s linux-image-generic 2021-08-21T14:52:53.080 INFO:teuthology.orchestra.run.smithi165.stdout:Package: linux-image-generic 2021-08-21T14:52:53.080 INFO:teuthology.orchestra.run.smithi165.stdout:Status: install ok installed 2021-08-21T14:52:53.080 INFO:teuthology.orchestra.run.smithi165.stdout:Priority: optional 2021-08-21T14:52:53.081 INFO:teuthology.orchestra.run.smithi165.stdout:Section: kernel 2021-08-21T14:52:53.081 INFO:teuthology.orchestra.run.smithi165.stdout:Installed-Size: 18 2021-08-21T14:52:53.081 INFO:teuthology.orchestra.run.smithi165.stdout:Maintainer: Ubuntu Kernel Team 2021-08-21T14:52:53.081 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: amd64 2021-08-21T14:52:53.081 INFO:teuthology.orchestra.run.smithi165.stdout:Source: linux-meta 2021-08-21T14:52:53.082 INFO:teuthology.orchestra.run.smithi165.stdout:Version: 5.4.0.81.85 2021-08-21T14:52:53.082 INFO:teuthology.orchestra.run.smithi165.stdout:Provides: virtualbox-guest-modules (= 6.1.16-dfsg-6~ubuntu1.20.04.2), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.10) 2021-08-21T14:52:53.082 INFO:teuthology.orchestra.run.smithi165.stdout:Depends: linux-image-5.4.0-81-generic, linux-modules-extra-5.4.0-81-generic, linux-firmware, intel-microcode, amd64-microcode 2021-08-21T14:52:53.082 INFO:teuthology.orchestra.run.smithi165.stdout:Recommends: thermald 2021-08-21T14:52:53.083 INFO:teuthology.orchestra.run.smithi165.stdout:Description: Generic Linux kernel image 2021-08-21T14:52:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: This package will always depend on the latest generic kernel image 2021-08-21T14:52:53.083 INFO:teuthology.orchestra.run.smithi165.stdout: available. 2021-08-21T14:52:53.084 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2021-08-21T14:52:53.090 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.qsrNVbJYfk 2021-08-21T14:52:53.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo cp /boot/grub/grub.cfg /tmp/tmp.qsrNVbJYfk 2021-08-21T14:52:53.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.qsrNVbJYfk 2021-08-21T14:52:53.198 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.qsrNVbJYfk is 8KB 2021-08-21T14:52:53.212 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.qsrNVbJYfk 2021-08-21T14:52:53.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-08-21T14:52:53.279 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:52:53.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-08-21T14:52:53.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-08-21T14:52:53.320 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-81-generic 2021-08-21T14:52:53.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo update-grub 2021-08-21T14:52:54.654 INFO:teuthology.orchestra.run.smithi165.stderr:Sourcing file `/etc/default/grub' 2021-08-21T14:52:54.656 INFO:teuthology.orchestra.run.smithi165.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2021-08-21T14:52:54.669 INFO:teuthology.orchestra.run.smithi165.stderr:Generating grub configuration file ... 2021-08-21T14:52:54.942 INFO:teuthology.orchestra.run.smithi165.stderr:Found linux image: /boot/vmlinuz-5.4.0-81-generic 2021-08-21T14:52:54.953 INFO:teuthology.orchestra.run.smithi165.stderr:Found initrd image: /boot/initrd.img-5.4.0-81-generic 2021-08-21T14:52:55.279 INFO:teuthology.orchestra.run.smithi165.stderr:Found linux image: /boot/vmlinuz-5.4.0-66-generic 2021-08-21T14:52:55.291 INFO:teuthology.orchestra.run.smithi165.stderr:Found initrd image: /boot/initrd.img-5.4.0-66-generic 2021-08-21T14:52:56.700 INFO:teuthology.orchestra.run.smithi165.stderr:done 2021-08-21T14:52:56.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo shutdown -r now 2021-08-21T14:53:26.736 INFO:teuthology.misc:Re-opening connections... 2021-08-21T14:53:26.737 INFO:teuthology.misc:trying to connect to ubuntu@smithi165.front.sepia.ceph.com 2021-08-21T14:53:26.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-21T14:53:26.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:54:01.419 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2021-08-21T14:54:01.420 DEBUG:teuthology.misc:waited 34.68363857269287 2021-08-21T14:54:02.421 INFO:teuthology.misc:trying to connect to ubuntu@smithi165.front.sepia.ceph.com 2021-08-21T14:54:02.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-21T14:54:02.422 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:54:05.487 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2021-08-21T14:54:05.487 DEBUG:teuthology.misc:waited 38.75060510635376 2021-08-21T14:54:06.489 INFO:teuthology.misc:trying to connect to ubuntu@smithi165.front.sepia.ceph.com 2021-08-21T14:54:06.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-21T14:54:06.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:54:08.555 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2021-08-21T14:54:08.555 DEBUG:teuthology.misc:waited 41.8185338973999 2021-08-21T14:54:09.557 INFO:teuthology.misc:trying to connect to ubuntu@smithi165.front.sepia.ceph.com 2021-08-21T14:54:09.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-21T14:54:09.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:54:41.582 DEBUG:teuthology.orchestra.run.smithi165:> true 2021-08-21T14:54:42.681 DEBUG:teuthology.misc:waited 75.94435954093933 2021-08-21T14:54:43.683 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2021-08-21T14:54:43.683 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-81-generic"... 2021-08-21T14:54:43.684 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2021-08-21T14:54:43.690 INFO:teuthology.orchestra.run.smithi165.stdout:5.4.0-81-generic 2021-08-21T14:54:43.690 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-81-generic vs 5.4.0-81-generic 2021-08-21T14:54:43.691 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-08-21T14:54:43.691 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-08-21T14:54:44.692 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-08-21T14:54:44.693 DEBUG:teuthology.orchestra.run.smithi165:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-08-21T14:54:44.761 INFO:teuthology.orchestra.run.smithi165.stdout:ttyS1 2021-08-21T14:54:44.789 INFO:teuthology.run_tasks:Running task internal.base... 2021-08-21T14:54:44.801 INFO:teuthology.task.internal:Creating test directory... 2021-08-21T14:54:44.801 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-08-21T14:54:44.809 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-08-21T14:54:44.849 INFO:teuthology.run_tasks:Running task internal.archive... 2021-08-21T14:54:44.900 INFO:teuthology.task.internal:Creating archive directory... 2021-08-21T14:54:44.900 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-08-21T14:54:44.924 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-08-21T14:54:44.950 INFO:teuthology.task.internal:Enabling coredump saving... 2021-08-21T14:54:44.951 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-08-21T14:54:44.972 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:54:44.981 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:54:44.983 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-08-21T14:54:45.000 INFO:teuthology.task.internal:Configuring sudo... 2021-08-21T14:54:45.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-08-21T14:54:45.021 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-08-21T14:54:45.058 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-08-21T14:54:45.059 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-08-21T14:54:45.074 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-08-21T14:54:45.082 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-08-21T14:54:45.088 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:54:45.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-08-21T14:54:45.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2021-08-21T14:54:45.198 INFO:teuthology.run_tasks:Running task internal.timer... 2021-08-21T14:54:45.206 INFO:teuthology.task.internal:Starting timer... 2021-08-21T14:54:45.207 INFO:teuthology.run_tasks:Running task pcp... 2021-08-21T14:54:45.273 INFO:teuthology.run_tasks:Running task selinux... 2021-08-21T14:54:45.324 DEBUG:teuthology.task.selinux:Excluding smithi165: OS 'ubuntu' does not support SELinux 2021-08-21T14:54:45.324 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-08-21T14:54:45.325 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-08-21T14:54:45.325 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-08-21T14:54:45.325 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-08-21T14:54:45.369 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-08-21T14:54:45.370 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-08-21T14:54:45.394 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-08-21T14:54:45.396 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi165.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-08-21T14:54:45.776 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-08-21T14:54:46.554 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-08-21T14:54:49.503 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-08-21T14:54:49.525 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:49 +0000 (0:00:00.039) 0:00:00.039 ******* 2021-08-21T14:54:51.668 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-08-21T14:54:51.670 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 20.04 on host smithi165.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi165.front.sepia.ceph.com] 2021-08-21T14:54:51.722 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:51 +0000 (0:00:02.196) 0:00:02.235 ******* 2021-08-21T14:54:51.946 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-08-21T14:54:51.947 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-08-21T14:54:51.974 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:51 +0000 (0:00:00.251) 0:00:02.487 ******* 2021-08-21T14:54:52.321 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-08-21T14:54:52.323 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-08-21T14:54:52.345 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:52 +0000 (0:00:00.371) 0:00:02.858 ******* 2021-08-21T14:54:52.403 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-08-21T14:54:52.405 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:54:52.426 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:52 +0000 (0:00:00.081) 0:00:02.940 ******* 2021-08-21T14:54:52.648 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-08-21T14:54:52.650 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-08-21T14:54:52.672 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:52 +0000 (0:00:00.245) 0:00:03.185 ******* 2021-08-21T14:54:53.383 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-08-21T14:54:53.385 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/cephlab_sudo", "size": 182, "state": "file", "uid": 0} 2021-08-21T14:54:53.406 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:53 +0000 (0:00:00.734) 0:00:03.920 ******* 2021-08-21T14:54:53.747 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-08-21T14:54:53.749 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-08-21T14:54:53.750 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-08-21T14:54:53.816 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-08-21T14:54:53.817 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-08-21T14:54:53.831 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Saturday 21 August 2021 14:54:53 +0000 (0:00:00.424) 0:00:04.345 ******* 2021-08-21T14:54:54.540 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] 2021-08-21T14:54:54.560 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** 2021-08-21T14:54:54.562 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:54 +0000 (0:00:00.728) 0:00:05.074 ******* 2021-08-21T14:54:54.586 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-08-21T14:54:54.592 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-08-21T14:54:54.761 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:54 +0000 (0:00:00.200) 0:00:05.275 ******* 2021-08-21T14:54:55.618 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi165.front.sepia.ceph.com] 2021-08-21T14:54:55.672 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-08-21T14:54:55.673 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:55 +0000 (0:00:00.910) 0:00:06.185 ******* 2021-08-21T14:54:56.271 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-08-21T14:54:56.273 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.159873", "end": "2021-08-21 14:54:56.250105", "rc": 0, "start": "2021-08-21 14:54:56.090232", "stderr": "", "stderr_lines": [], "stdout": "7f6ec18127eb11437e244122daa7109a4352270a", "stdout_lines": ["7f6ec18127eb11437e244122daa7109a4352270a"]} 2021-08-21T14:54:56.311 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:56 +0000 (0:00:00.639) 0:00:06.824 ******* 2021-08-21T14:54:56.536 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-08-21T14:54:56.538 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.087456", "end": "2021-08-21 14:54:56.118042", "failed_when_result": false, "rc": 0, "start": "2021-08-21 14:54:56.030586", "stderr": "", "stderr_lines": [], "stdout": "c29274177b79f72740fc714fa28b55400cd0737f", "stdout_lines": ["c29274177b79f72740fc714fa28b55400cd0737f"]} 2021-08-21T14:54:56.560 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:56 +0000 (0:00:00.249) 0:00:07.074 ******* 2021-08-21T14:54:56.620 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-08-21T14:54:56.622 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:54:56.644 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:56 +0000 (0:00:00.083) 0:00:07.157 ******* 2021-08-21T14:54:56.711 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-08-21T14:54:56.713 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:54:56.735 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:56 +0000 (0:00:00.091) 0:00:07.249 ******* 2021-08-21T14:54:56.803 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-08-21T14:54:56.805 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:54:56.826 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:56 +0000 (0:00:00.090) 0:00:07.340 ******* 2021-08-21T14:54:56.894 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:54:56.918 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:56 +0000 (0:00:00.092) 0:00:07.432 ******* 2021-08-21T14:54:56.986 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-08-21T14:54:56.988 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:54:57.010 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:57 +0000 (0:00:00.092) 0:00:07.524 ******* 2021-08-21T14:54:57.184 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-08-21T14:54:57.185 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-08-21T14:54:57.207 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:54:57 +0000 (0:00:00.197) 0:00:07.721 ******* 2021-08-21T14:54:57.442 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-08-21T14:54:57.444 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-08-21T14:54:57.588 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-08-21T14:54:57.723 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-08-21T14:54:57.868 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-08-21T14:54:58.025 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-08-21T14:54:58.160 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-08-21T14:54:58.307 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-08-21T14:54:58.452 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-08-21T14:54:58.598 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-08-21T14:54:58.736 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-08-21T14:54:58.872 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-08-21T14:54:59.018 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-08-21T14:54:59.164 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-08-21T14:54:59.310 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-08-21T14:54:59.455 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-08-21T14:54:59.589 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-08-21T14:54:59.746 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-08-21T14:54:59.881 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-08-21T14:55:00.029 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-08-21T14:55:00.174 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-08-21T14:55:00.319 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-08-21T14:55:00.487 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-08-21T14:55:00.623 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-08-21T14:55:00.770 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-08-21T14:55:00.915 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-08-21T14:55:01.051 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-08-21T14:55:01.196 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-08-21T14:55:01.343 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-08-21T14:55:01.489 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-08-21T14:55:01.625 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-08-21T14:55:01.780 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-08-21T14:55:01.915 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-08-21T14:55:02.060 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-08-21T14:55:02.206 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-08-21T14:55:02.343 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-08-21T14:55:02.489 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-08-21T14:55:02.636 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-08-21T14:55:02.782 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-08-21T14:55:02.928 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-08-21T14:55:03.074 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-08-21T14:55:03.220 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-08-21T14:55:03.356 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-08-21T14:55:03.502 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-08-21T14:55:03.648 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-08-21T14:55:03.793 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-08-21T14:55:03.940 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-08-21T14:55:04.086 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-08-21T14:55:04.233 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-08-21T14:55:04.379 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-08-21T14:55:04.525 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-08-21T14:55:04.670 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-08-21T14:55:04.817 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-08-21T14:55:04.963 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-08-21T14:55:05.108 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-08-21T14:55:05.254 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-08-21T14:55:05.389 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-08-21T14:55:05.534 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-08-21T14:55:05.680 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-08-21T14:55:05.816 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-08-21T14:55:05.971 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-08-21T14:55:06.106 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-08-21T14:55:06.257 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-08-21T14:55:06.392 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-08-21T14:55:06.538 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-08-21T14:55:06.686 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-08-21T14:55:06.843 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-08-21T14:55:06.989 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-08-21T14:55:07.125 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-08-21T14:55:07.272 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-08-21T14:55:07.418 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-08-21T14:55:07.561 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-08-21T14:55:07.707 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-08-21T14:55:07.852 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-08-21T14:55:07.997 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-08-21T14:55:08.152 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-08-21T14:55:08.298 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-08-21T14:55:08.444 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-08-21T14:55:08.588 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-08-21T14:55:08.733 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-08-21T14:55:08.868 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-08-21T14:55:09.014 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-08-21T14:55:09.159 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-08-21T14:55:09.306 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-08-21T14:55:09.451 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-08-21T14:55:09.596 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-08-21T14:55:09.741 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-08-21T14:55:09.878 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-08-21T14:55:10.024 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-08-21T14:55:10.170 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-08-21T14:55:10.316 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-08-21T14:55:10.451 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-08-21T14:55:10.596 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-08-21T14:55:10.742 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-08-21T14:55:10.876 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-08-21T14:55:11.022 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-08-21T14:55:11.167 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-08-21T14:55:11.313 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-08-21T14:55:11.458 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-08-21T14:55:11.593 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-08-21T14:55:11.740 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-08-21T14:55:11.885 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-08-21T14:55:12.031 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-08-21T14:55:12.177 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-08-21T14:55:12.323 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-08-21T14:55:12.468 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-08-21T14:55:12.613 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-08-21T14:55:12.748 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-08-21T14:55:12.904 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-08-21T14:55:13.039 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-08-21T14:55:13.185 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-08-21T14:55:13.330 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-08-21T14:55:13.475 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-08-21T14:55:13.620 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-08-21T14:55:13.765 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-08-21T14:55:13.901 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-08-21T14:55:14.058 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-08-21T14:55:14.193 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-08-21T14:55:14.339 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-08-21T14:55:14.484 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-08-21T14:55:14.629 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-08-21T14:55:14.765 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-08-21T14:55:14.911 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-08-21T14:55:15.046 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-08-21T14:55:15.194 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-08-21T14:55:15.339 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-08-21T14:55:15.474 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-08-21T14:55:15.620 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-08-21T14:55:15.766 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-08-21T14:55:15.912 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-08-21T14:55:16.057 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-08-21T14:55:16.205 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-08-21T14:55:16.350 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-08-21T14:55:16.496 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-08-21T14:55:16.636 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-08-21T14:55:16.782 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-08-21T14:55:16.929 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-08-21T14:55:17.066 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-08-21T14:55:17.212 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-08-21T14:55:17.359 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10103, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 10102} 2021-08-21T14:55:17.515 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10104, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "move_home": false, "name": "amctagga", "shell": "/bin/bash", "state": "present", "uid": 10103} 2021-08-21T14:55:17.651 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10105, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "move_home": false, "name": "kevinz", "shell": "/bin/bash", "state": "present", "uid": 10104} 2021-08-21T14:55:17.798 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10106, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "move_home": false, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "uid": 10105} 2021-08-21T14:55:17.934 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10107, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "move_home": false, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "uid": 10106} 2021-08-21T14:55:18.081 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10108, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "move_home": false, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "uid": 10107} 2021-08-21T14:55:18.220 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10109, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "move_home": false, "name": "lflores", "shell": "/bin/bash", "state": "present", "uid": 10108} 2021-08-21T14:55:18.364 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10110, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "move_home": false, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "uid": 10109} 2021-08-21T14:55:18.512 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10111, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "move_home": false, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "uid": 10110} 2021-08-21T14:55:18.657 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10112, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "move_home": false, "name": "luisrico", "shell": "/bin/bash", "state": "present", "uid": 10111} 2021-08-21T14:55:18.793 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10113, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, "move_home": false, "name": "matan", "shell": "/bin/bash", "state": "present", "uid": 10112} 2021-08-21T14:55:18.938 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10114, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "move_home": false, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "uid": 10113} 2021-08-21T14:55:19.085 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10115, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "move_home": false, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "uid": 10114} 2021-08-21T14:55:19.232 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10116, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "move_home": false, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "uid": 10115} 2021-08-21T14:55:19.378 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10117, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "move_home": false, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "uid": 10116} 2021-08-21T14:55:19.513 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10118, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "move_home": false, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "uid": 10117} 2021-08-21T14:55:19.666 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10119, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "move_home": false, "name": "melissali", "shell": "/bin/bash", "state": "present", "uid": 10118} 2021-08-21T14:55:20.409 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-08-21T14:55:20.855 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10121, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-08-21T14:55:21.320 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10122, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10121} 2021-08-21T14:55:21.447 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:21 +0000 (0:00:24.239) 0:00:31.960 ******* 2021-08-21T14:55:21.508 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-08-21T14:55:21.531 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:21 +0000 (0:00:00.084) 0:00:32.044 ******* 2021-08-21T14:55:21.621 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-08-21T14:55:21.623 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-08-21T14:55:21.625 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-08-21T14:55:21.626 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-08-21T14:55:21.629 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-08-21T14:55:21.630 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9 2021-08-21T14:55:21.632 INFO:teuthology.task.ansible.out:b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", " 2021-08-21T14:55:21.634 INFO:teuthology.task.ansible.out:ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn" 2021-08-21T14:55:21.634 INFO:teuthology.task.ansible.out:: "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 2021-08-21T14:55:21.636 INFO:teuthology.task.ansible.out:f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": 2021-08-21T14:55:21.638 INFO:teuthology.task.ansible.out:"alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon 2021-08-21T14:55:21.639 INFO:teuthology.task.ansible.out:", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"nam 2021-08-21T14:55:21.641 INFO:teuthology.task.ansible.out:e": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e104 2021-08-21T14:55:21.645 INFO:teuthology.task.ansible.out:6655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWC 2021-08-21T14:55:21.650 INFO:teuthology.task.ansible.out:XLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf94 2021-08-21T14:55:21.651 INFO:teuthology.task.ansible.out:60980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+ 2021-08-21T14:55:21.653 INFO:teuthology.task.ansible.out:GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}]}, "changed": false} 2021-08-21T14:55:21.671 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:21 +0000 (0:00:00.139) 0:00:32.184 ******* 2021-08-21T14:55:22.531 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [smithi165.front.sepia.ceph.com] => {"after": "7f6ec18127eb11437e244122daa7109a4352270a", "attempts": 1, "before": "7f6ec18127eb11437e244122daa7109a4352270a", "changed": false, "remote_url_changed": false} 2021-08-21T14:55:22.555 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:22 +0000 (0:00:00.884) 0:00:33.069 ******* 2021-08-21T14:55:23.048 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-08-21T14:55:23.050 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-08-21T14:55:23.185 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-08-21T14:55:23.321 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-08-21T14:55:23.457 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-08-21T14:55:23.459 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-08-21T14:55:23.594 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-08-21T14:55:23.719 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-08-21T14:55:23.721 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-08-21T14:55:23.854 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-08-21T14:55:23.980 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-08-21T14:55:24.115 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-08-21T14:55:24.127 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:24.251 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-08-21T14:55:24.253 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-08-21T14:55:24.386 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-08-21T14:55:24.388 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:24.398 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:24.533 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-08-21T14:55:24.668 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-08-21T14:55:24.670 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-08-21T14:55:24.793 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-08-21T14:55:24.929 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-08-21T14:55:24.930 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-08-21T14:55:25.056 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-08-21T14:55:25.191 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-08-21T14:55:25.317 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-08-21T14:55:25.452 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-08-21T14:55:25.454 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-08-21T14:55:25.576 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-08-21T14:55:25.578 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-08-21T14:55:25.713 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-08-21T14:55:25.839 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-08-21T14:55:25.985 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlI 2021-08-21T14:55:25.987 INFO:teuthology.task.ansible.out:Qv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-08-21T14:55:26.120 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-08-21T14:55:26.245 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-08-21T14:55:26.381 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-08-21T14:55:26.515 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-08-21T14:55:26.641 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-08-21T14:55:26.777 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-08-21T14:55:26.778 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-08-21T14:55:26.922 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-08-21T14:55:27.057 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-08-21T14:55:27.182 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-08-21T14:55:27.317 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-08-21T14:55:27.451 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-08-21T14:55:27.577 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-08-21T14:55:27.711 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-08-21T14:55:27.846 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-08-21T14:55:27.971 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-08-21T14:55:27.973 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-08-21T14:55:28.107 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-08-21T14:55:28.109 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-08-21T14:55:28.233 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-08-21T14:55:28.379 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-08-21T14:55:28.504 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-08-21T14:55:28.644 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-08-21T14:55:28.646 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-08-21T14:55:28.779 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-08-21T14:55:28.781 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-08-21T14:55:28.782 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-08-21T14:55:28.916 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-08-21T14:55:29.052 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-08-21T14:55:29.177 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-08-21T14:55:29.315 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-08-21T14:55:29.441 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-08-21T14:55:29.578 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-08-21T14:55:29.580 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-08-21T14:55:29.718 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-08-21T14:55:29.854 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-08-21T14:55:29.990 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-08-21T14:55:29.992 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-08-21T14:55:30.118 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-08-21T14:55:30.257 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-08-21T14:55:30.382 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-08-21T14:55:30.384 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-08-21T14:55:30.519 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-08-21T14:55:30.656 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-08-21T14:55:30.782 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-08-21T14:55:30.907 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-08-21T14:55:31.053 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-08-21T14:55:31.188 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-08-21T14:55:31.345 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-08-21T14:55:31.473 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-08-21T14:55:31.610 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-08-21T14:55:31.736 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-08-21T14:55:31.873 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-08-21T14:55:31.999 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-08-21T14:55:32.136 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-08-21T14:55:32.263 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-08-21T14:55:32.399 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-08-21T14:55:32.547 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-08-21T14:55:32.674 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-08-21T14:55:32.814 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-08-21T14:55:32.944 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-08-21T14:55:33.080 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-08-21T14:55:33.220 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-08-21T14:55:33.222 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-08-21T14:55:33.369 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-08-21T14:55:33.506 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-08-21T14:55:33.644 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-08-21T14:55:33.801 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-08-21T14:55:33.930 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-08-21T14:55:33.932 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-08-21T14:55:34.075 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-08-21T14:55:34.077 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-08-21T14:55:34.214 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-08-21T14:55:34.340 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-08-21T14:55:34.477 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-08-21T14:55:34.604 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-08-21T14:55:34.741 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-08-21T14:55:34.877 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-08-21T14:55:35.014 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-08-21T14:55:35.141 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-08-21T14:55:35.142 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-08-21T14:55:35.286 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-08-21T14:55:35.422 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-08-21T14:55:35.558 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-08-21T14:55:35.695 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-08-21T14:55:35.833 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-08-21T14:55:35.967 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-08-21T14:55:36.093 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-08-21T14:55:36.095 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-08-21T14:55:36.229 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-08-21T14:55:36.367 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-08-21T14:55:36.492 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-08-21T14:55:36.627 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-08-21T14:55:36.754 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-08-21T14:55:36.888 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-08-21T14:55:37.012 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-08-21T14:55:37.148 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-08-21T14:55:37.282 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-08-21T14:55:37.418 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-08-21T14:55:37.564 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-08-21T14:55:37.690 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-08-21T14:55:37.825 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-08-21T14:55:37.950 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-08-21T14:55:38.086 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-08-21T14:55:38.216 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-08-21T14:55:38.352 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-08-21T14:55:38.478 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-08-21T14:55:38.613 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-08-21T14:55:38.615 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-08-21T14:55:38.741 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-08-21T14:55:38.882 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-08-21T14:55:39.020 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-08-21T14:55:39.163 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-08-21T14:55:39.309 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-08-21T14:55:39.457 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-08-21T14:55:39.594 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-08-21T14:55:39.719 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-08-21T14:55:39.858 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-08-21T14:55:39.987 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-08-21T14:55:40.134 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-08-21T14:55:40.260 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-08-21T14:55:40.395 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-08-21T14:55:40.530 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-08-21T14:55:40.655 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-08-21T14:55:40.793 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-08-21T14:55:40.939 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-08-21T14:55:40.941 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-08-21T14:55:41.064 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-08-21T14:55:41.211 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-08-21T14:55:41.347 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-08-21T14:55:41.472 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-08-21T14:55:41.609 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-08-21T14:55:41.744 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-08-21T14:55:41.883 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-08-21T14:55:42.007 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-08-21T14:55:42.145 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-08-21T14:55:42.270 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-08-21T14:55:42.405 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-08-21T14:55:42.531 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-08-21T14:55:42.667 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-08-21T14:55:42.803 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-08-21T14:55:42.940 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-08-21T14:55:43.077 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-08-21T14:55:43.204 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-08-21T14:55:43.339 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-08-21T14:55:43.474 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-08-21T14:55:43.599 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-08-21T14:55:43.734 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-08-21T14:55:43.843 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:43 +0000 (0:00:21.287) 0:00:54.356 ******* 2021-08-21T14:55:43.932 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-08-21T14:55:43.934 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:43.944 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:43.955 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:43.957 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:43.967 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:43.977 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:43.988 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:43.990 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.135 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-08-21T14:55:44.137 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.147 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.272 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-08-21T14:55:44.407 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-08-21T14:55:44.409 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.418 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.420 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.429 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.440 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.442 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.454 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.464 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.466 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.475 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.486 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.497 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.497 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.510 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.519 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.530 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.541 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.543 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.552 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.562 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.564 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.574 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.575 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.585 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.597 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.607 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.609 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.619 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.620 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.630 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.641 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.642 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.652 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.653 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.663 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.674 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.675 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.686 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.696 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.707 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.709 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.719 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.720 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.730 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.741 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.752 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.754 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.763 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.765 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.775 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.785 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.787 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.797 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.808 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.819 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.821 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.830 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.832 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.841 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.843 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.853 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.863 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.865 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.874 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.876 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.886 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.887 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.897 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.908 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.909 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.919 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.920 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.932 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.941 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.943 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.953 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.954 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.964 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.975 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.977 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.987 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.989 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:44.998 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.009 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.010 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.020 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.030 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.032 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.042 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.044 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.053 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.064 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.066 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.075 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.077 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.086 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.097 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.108 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.110 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.119 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.121 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.131 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.142 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.144 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.153 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.155 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.164 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.175 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.177 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.186 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.188 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.197 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.199 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.209 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.220 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.222 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.231 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.233 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.242 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.253 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.255 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.265 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.266 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.276 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.287 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.289 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.298 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.300 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.310 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.320 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.322 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.332 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.334 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.343 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.354 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.356 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.367 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.369 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.420 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:45 +0000 (0:00:01.577) 0:00:55.933 ******* 2021-08-21T14:55:45.487 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-08-21T14:55:45.489 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:45.511 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:45 +0000 (0:00:00.090) 0:00:56.024 ******* 2021-08-21T14:55:45.734 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-08-21T14:55:45.736 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-08-21T14:55:45.860 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-08-21T14:55:45.996 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-08-21T14:55:46.132 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-08-21T14:55:46.267 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-08-21T14:55:46.403 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-08-21T14:55:46.540 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-08-21T14:55:46.666 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-08-21T14:55:46.802 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-08-21T14:55:46.940 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-08-21T14:55:47.078 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-08-21T14:55:47.204 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-08-21T14:55:47.342 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-08-21T14:55:47.477 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-08-21T14:55:47.612 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-08-21T14:55:47.748 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-08-21T14:55:47.875 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-08-21T14:55:48.020 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-08-21T14:55:48.156 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-08-21T14:55:48.280 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-08-21T14:55:48.428 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-08-21T14:55:48.564 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-08-21T14:55:48.699 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-08-21T14:55:48.834 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-08-21T14:55:48.970 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-08-21T14:55:49.105 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-08-21T14:55:49.251 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-08-21T14:55:49.386 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-08-21T14:55:49.521 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-08-21T14:55:49.655 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-08-21T14:55:49.790 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-08-21T14:55:49.925 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-08-21T14:55:50.061 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-08-21T14:55:50.196 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-08-21T14:55:50.331 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-08-21T14:55:50.466 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-08-21T14:55:50.601 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-08-21T14:55:50.726 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-08-21T14:55:50.861 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-08-21T14:55:50.996 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-08-21T14:55:51.131 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-08-21T14:55:51.266 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-08-21T14:55:51.391 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-08-21T14:55:51.526 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-08-21T14:55:51.661 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-08-21T14:55:51.797 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-08-21T14:55:51.923 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-08-21T14:55:52.060 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-08-21T14:55:52.196 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-08-21T14:55:52.239 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:52 +0000 (0:00:06.728) 0:01:02.752 ******* 2021-08-21T14:55:53.189 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-08-21T14:55:53.191 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "checksum": "39e5632aabd93a7648a3940c07007a14afe2d320", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "1f362c6963bf1f0d83c5c22328e42c02", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1629557752.3195567-1469-196442685238005/source", "state": "file", "uid": 0} 2021-08-21T14:55:53.222 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:53 +0000 (0:00:00.982) 0:01:03.735 ******* 2021-08-21T14:55:53.291 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-08-21T14:55:53.293 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => { "msg": "Host smithi165.front.sepia.ceph.com is running Ubuntu 20.04 (focal)" } 2021-08-21T14:55:53.315 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:53 +0000 (0:00:00.092) 0:01:03.828 ******* 2021-08-21T14:55:53.393 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-08-21T14:55:53.395 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-08-21T14:55:53.419 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:53 +0000 (0:00:00.104) 0:01:03.933 ******* 2021-08-21T14:55:53.487 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-08-21T14:55:53.489 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:53.511 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:53 +0000 (0:00:00.092) 0:01:04.025 ******* 2021-08-21T14:55:53.601 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-08-21T14:55:53.603 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:53.625 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:53 +0000 (0:00:00.113) 0:01:04.138 ******* 2021-08-21T14:55:53.694 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-08-21T14:55:53.695 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:53.718 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:53 +0000 (0:00:00.093) 0:01:04.231 ******* 2021-08-21T14:55:53.786 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-08-21T14:55:53.788 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:53.810 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:53 +0000 (0:00:00.092) 0:01:04.323 ******* 2021-08-21T14:55:53.879 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-08-21T14:55:53.881 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:53.904 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:53 +0000 (0:00:00.092) 0:01:04.416 ******* 2021-08-21T14:55:53.972 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-08-21T14:55:53.974 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:53.995 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:53 +0000 (0:00:00.093) 0:01:04.509 ******* 2021-08-21T14:55:54.063 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-08-21T14:55:54.065 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:54.086 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:54 +0000 (0:00:00.090) 0:01:04.599 ******* 2021-08-21T14:55:54.174 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:54.197 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:54 +0000 (0:00:00.111) 0:01:04.711 ******* 2021-08-21T14:55:54.276 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-08-21T14:55:54.278 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi165.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-08-21T14:55:54.304 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:54 +0000 (0:00:00.106) 0:01:04.818 ******* 2021-08-21T14:55:54.374 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:54.397 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:54 +0000 (0:00:00.093) 0:01:04.911 ******* 2021-08-21T14:55:54.457 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-08-21T14:55:54.459 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:54.481 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:54 +0000 (0:00:00.083) 0:01:04.994 ******* 2021-08-21T14:55:54.550 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:54.573 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:54 +0000 (0:00:00.092) 0:01:05.087 ******* 2021-08-21T14:55:54.642 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-08-21T14:55:54.643 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-08-21T14:55:54.666 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:54 +0000 (0:00:00.092) 0:01:05.179 ******* 2021-08-21T14:55:54.734 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:54.757 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:54 +0000 (0:00:00.091) 0:01:05.271 ******* 2021-08-21T14:55:54.815 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-08-21T14:55:54.817 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-08-21T14:55:54.838 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:54 +0000 (0:00:00.081) 0:01:05.352 ******* 2021-08-21T14:55:54.919 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-08-21T14:55:54.921 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:54.943 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:54 +0000 (0:00:00.104) 0:01:05.456 ******* 2021-08-21T14:55:55.014 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:55.038 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:55 +0000 (0:00:00.094) 0:01:05.551 ******* 2021-08-21T14:55:55.106 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:55.130 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:55 +0000 (0:00:00.091) 0:01:05.643 ******* 2021-08-21T14:55:55.200 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:55.223 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:55 +0000 (0:00:00.093) 0:01:05.737 ******* 2021-08-21T14:55:55.292 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-08-21T14:55:55.294 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-08-21T14:55:55.316 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:55 +0000 (0:00:00.092) 0:01:05.829 ******* 2021-08-21T14:55:55.406 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-08-21T14:55:55.409 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:55.433 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:55 +0000 (0:00:00.117) 0:01:05.947 ******* 2021-08-21T14:55:55.501 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-08-21T14:55:55.503 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:55.525 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:55 +0000 (0:00:00.091) 0:01:06.038 ******* 2021-08-21T14:55:55.594 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-08-21T14:55:55.596 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:55.630 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:55 +0000 (0:00:00.105) 0:01:06.144 ******* 2021-08-21T14:55:55.701 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:55.724 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:55 +0000 (0:00:00.093) 0:01:06.238 ******* 2021-08-21T14:55:55.792 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-08-21T14:55:55.794 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:55.816 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:55 +0000 (0:00:00.092) 0:01:06.330 ******* 2021-08-21T14:55:55.884 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-08-21T14:55:55.886 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:55.908 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:55 +0000 (0:00:00.091) 0:01:06.421 ******* 2021-08-21T14:55:55.987 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-08-21T14:55:55.988 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-08-21T14:55:56.010 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:56 +0000 (0:00:00.102) 0:01:06.524 ******* 2021-08-21T14:55:56.078 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-08-21T14:55:56.080 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:56.102 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:56 +0000 (0:00:00.091) 0:01:06.615 ******* 2021-08-21T14:55:56.171 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-08-21T14:55:56.173 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:56.194 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:56 +0000 (0:00:00.091) 0:01:06.707 ******* 2021-08-21T14:55:56.262 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:56.287 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:56 +0000 (0:00:00.092) 0:01:06.800 ******* 2021-08-21T14:55:56.367 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:56.368 INFO:teuthology.task.ansible.out: skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:56.393 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:56 +0000 (0:00:00.106) 0:01:06.906 ******* 2021-08-21T14:55:56.471 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-08-21T14:55:56.473 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-08-21T14:55:56.495 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:56 +0000 (0:00:00.102) 0:01:07.009 ******* 2021-08-21T14:55:56.563 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:56.590 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:56 +0000 (0:00:00.094) 0:01:07.104 ******* 2021-08-21T14:55:56.968 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-08-21T14:55:56.970 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.197176", "end": "2021-08-21 14:55:56.552232", "rc": 0, "start": "2021-08-21 14:55:56.355056", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-08-21T14:55:56.992 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:56 +0000 (0:00:00.401) 0:01:07.505 ******* 2021-08-21T14:55:57.061 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-08-21T14:55:57.062 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:57.085 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:57 +0000 (0:00:00.093) 0:01:07.599 ******* 2021-08-21T14:55:57.154 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-08-21T14:55:57.156 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:57.178 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:57 +0000 (0:00:00.092) 0:01:07.691 ******* 2021-08-21T14:55:57.879 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-08-21T14:55:57.881 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "name": "sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit sleep.target is masked.\"", "LoadState": "masked", "Names": "sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-08-21T14:55:58.057 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "name": "suspend.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "suspend.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "suspend.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit suspend.target is masked.\"", "LoadState": "masked", "Names": "suspend.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-08-21T14:55:58.234 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "name": "hibernate.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hibernate.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hibernate.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hibernate.target is masked.\"", "LoadState": "masked", "Names": "hibernate.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-08-21T14:55:58.410 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "name": "hybrid-sleep.target", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "AllowIsolate": "no", "AssertResult": "no", "AssertTimestampMonotonic": "0", "CanIsolate": "no", "CanReload": "no", "CanStart": "no", "CanStop": "yes", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "DefaultDependencies": "yes", "Description": "hybrid-sleep.target", "FailureAction": "none", "FragmentPath": "/dev/null", "Id": "hybrid-sleep.target", "IgnoreOnIsolate": "no", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "LoadError": "org.freedesktop.systemd1.UnitMasked \"Unit hybrid-sleep.target is masked.\"", "LoadState": "masked", "Names": "hybrid-sleep.target", "NeedDaemonReload": "no", "OnFailureJobMode": "replace", "Perpetual": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StateChangeTimestampMonotonic": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "Transient": "no", "UnitFileState": "masked"}} 2021-08-21T14:55:58.435 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:58 +0000 (0:00:01.257) 0:01:08.949 ******* 2021-08-21T14:55:58.503 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-08-21T14:55:58.505 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:58.526 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:58 +0000 (0:00:00.090) 0:01:09.039 ******* 2021-08-21T14:55:58.594 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-08-21T14:55:58.595 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:58.617 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:58 +0000 (0:00:00.090) 0:01:09.130 ******* 2021-08-21T14:55:58.684 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-08-21T14:55:58.686 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:58.710 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:58 +0000 (0:00:00.093) 0:01:09.223 ******* 2021-08-21T14:55:58.768 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:58.791 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:58 +0000 (0:00:00.080) 0:01:09.304 ******* 2021-08-21T14:55:58.858 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-08-21T14:55:58.860 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:55:58.881 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:58 +0000 (0:00:00.090) 0:01:09.395 ******* 2021-08-21T14:55:59.918 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-08-21T14:55:59.920 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.899097", "end": "2021-08-21 14:55:59.500472", "rc": 0, "start": "2021-08-21 14:55:58.601375", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:55:59.941 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:55:59 +0000 (0:00:01.060) 0:01:10.455 ******* 2021-08-21T14:56:00.020 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:00.043 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:00 +0000 (0:00:00.101) 0:01:10.557 ******* 2021-08-21T14:56:00.464 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-08-21T14:56:00.465 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.088885", "end": "2021-08-21 14:56:00.046011", "rc": 0, "start": "2021-08-21 14:55:59.957126", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:56:00.487 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:00 +0000 (0:00:00.443) 0:01:11.001 ******* 2021-08-21T14:56:04.221 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-08-21T14:56:04.231 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:56:04.245 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:04 +0000 (0:00:03.757) 0:01:14.758 ******* 2021-08-21T14:56:05.110 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-08-21T14:56:05.112 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:56:05.134 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:05 +0000 (0:00:00.889) 0:01:15.647 ******* 2021-08-21T14:56:05.347 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-08-21T14:56:05.348 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:05.494 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:05 +0000 (0:00:00.359) 0:01:16.006 ******* 2021-08-21T14:56:05.872 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-08-21T14:56:05.874 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-08-21T14:56:05.896 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:05 +0000 (0:00:00.403) 0:01:16.409 ******* 2021-08-21T14:56:06.626 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-08-21T14:56:06.628 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:56:06.650 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:06 +0000 (0:00:00.753) 0:01:17.163 ******* 2021-08-21T14:56:07.043 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi165.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-08-21T14:56:07.367 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-08-21T14:56:07.669 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-08-21T14:56:07.694 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:07 +0000 (0:00:01.044) 0:01:18.208 ******* 2021-08-21T14:56:07.836 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-08-21T14:56:07.859 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:07 +0000 (0:00:00.164) 0:01:18.372 ******* 2021-08-21T14:56:08.238 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-08-21T14:56:08.240 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-08-21T14:56:08.522 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-08-21T14:56:08.825 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3612, "state": "file", "uid": 0} 2021-08-21T14:56:08.849 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:08 +0000 (0:00:00.990) 0:01:19.363 ******* 2021-08-21T14:56:08.928 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-08-21T14:56:08.931 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-08-21T14:56:08.952 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:08 +0000 (0:00:00.102) 0:01:19.465 ******* 2021-08-21T14:56:09.020 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-08-21T14:56:09.022 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:09.045 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:09 +0000 (0:00:00.092) 0:01:19.558 ******* 2021-08-21T14:56:09.114 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-08-21T14:56:09.117 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-08-21T14:56:09.138 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:09 +0000 (0:00:00.093) 0:01:19.651 ******* 2021-08-21T14:56:09.217 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-08-21T14:56:09.219 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-08-21T14:56:09.242 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-08-21T14:56:09.250 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-08-21T14:56:09.263 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-08-21T14:56:09.285 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:09 +0000 (0:00:00.147) 0:01:19.799 ******* 2021-08-21T14:56:09.354 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-08-21T14:56:09.356 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:09.378 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:09 +0000 (0:00:00.092) 0:01:19.891 ******* 2021-08-21T14:56:10.098 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** ok: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:56:10.121 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:10 +0000 (0:00:00.743) 0:01:20.634 ******* 2021-08-21T14:56:10.495 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-08-21T14:56:10.497 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-08-21T14:56:10.518 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:10 +0000 (0:00:00.397) 0:01:21.032 ******* 2021-08-21T14:56:10.901 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-08-21T14:56:10.903 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-08-21T14:56:10.937 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:10 +0000 (0:00:00.418) 0:01:21.450 ******* 2021-08-21T14:56:11.142 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-08-21T14:56:11.144 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-08-21T14:56:11.259 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-08-21T14:56:11.286 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:11 +0000 (0:00:00.349) 0:01:21.800 ******* 2021-08-21T14:56:11.355 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-08-21T14:56:11.357 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:11.378 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:11 +0000 (0:00:00.092) 0:01:21.892 ******* 2021-08-21T14:56:11.447 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-08-21T14:56:11.449 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-08-21T14:56:11.472 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:11 +0000 (0:00:00.092) 0:01:21.985 ******* 2021-08-21T14:56:11.551 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-08-21T14:56:11.553 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:11.576 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:11 +0000 (0:00:00.103) 0:01:22.089 ******* 2021-08-21T14:56:11.969 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-08-21T14:56:11.971 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-08-21T14:56:11.993 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:11 +0000 (0:00:00.417) 0:01:22.506 ******* 2021-08-21T14:56:12.258 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-08-21 14:54:38 UTC", "ActiveEnterTimestampMonotonic": "13476861", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target -.mount nss-lookup.target time-sync.target system.slice systemd-tmpfiles-setup.service local-fs.target systemd-journald.socket var-run.mount sysinit.target remote-fs.target network.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-08-21 14:54:38 UTC", "AssertTimestampMonotonic": "13472478", "Before": "plymouth-quit.service getty@tty1.service xdm.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive" 2021-08-21T14:56:12.260 INFO:teuthology.task.ansible.out:, "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-08-21 14:54:38 UTC", "ConditionTimestampMonotonic": "13472477", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "845", "ExecMainStartTimestamp": "Sat 2021-08-21 14:54:38 UTC", "ExecMainStartTimestampMonotonic": "13476723", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Sat 2021-08-21 14:54:38 UTC] ; stop_time=[n/a] ; pid=845 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; flags= ; start_time=[Sat 2021-08-21 14:54:38 UTC] ; stop_time=[n/a] ; pid=845 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPostEx": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nagios/nrpe.pid ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKil 2021-08-21T14:56:12.262 INFO:teuthology.task.ansible.out:lSignal": "9", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-08-21 14:54:38 UTC", "InactiveExitTimestampMonotonic": "13476861", "InvocationID": "f7783606b5d3496ab73437929e42da63", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127777", "LimitNPROCSoft": "127777", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755 2021-08-21T14:56:12.263 INFO:teuthology.task.ansible.out:", "MainPID": "845", "MemoryAccounting": "yes", "MemoryCurrent": "1306624", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nagios/nrpe.pid", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-08-21 14:54:38 UTC", "StateChangeTimestampMonotonic": "1 2021-08-21T14:56:12.265 INFO:teuthology.task.ansible.out:3476861", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "38333", "TimeoutAbortUSec": "1min", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "110", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-08-21T14:56:12.284 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:12 +0000 (0:00:00.290) 0:01:22.797 ******* 2021-08-21T14:56:12.343 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-08-21T14:56:12.345 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:12.368 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:12 +0000 (0:00:00.085) 0:01:22.882 ******* 2021-08-21T14:56:12.451 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-08-21T14:56:12.453 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:12.475 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:12 +0000 (0:00:00.106) 0:01:22.989 ******* 2021-08-21T14:56:12.556 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-08-21T14:56:12.558 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:12.580 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:12 +0000 (0:00:00.104) 0:01:23.093 ******* 2021-08-21T14:56:12.651 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:12.680 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:12 +0000 (0:00:00.100) 0:01:23.193 ******* 2021-08-21T14:56:12.761 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-08-21T14:56:12.763 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:12.785 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:12 +0000 (0:00:00.104) 0:01:23.298 ******* 2021-08-21T14:56:12.862 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-08-21T14:56:12.864 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:12.886 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:12 +0000 (0:00:00.101) 0:01:23.399 ******* 2021-08-21T14:56:12.964 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-08-21T14:56:12.966 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:12.988 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:12 +0000 (0:00:00.101) 0:01:23.501 ******* 2021-08-21T14:56:13.066 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-08-21T14:56:13.068 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:13.089 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:13 +0000 (0:00:00.101) 0:01:23.603 ******* 2021-08-21T14:56:13.159 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:13.187 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:13 +0000 (0:00:00.097) 0:01:23.700 ******* 2021-08-21T14:56:13.266 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-08-21T14:56:13.308 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Saturday 21 August 2021 14:56:13 +0000 (0:00:00.103) 0:01:23.803 ******* 2021-08-21T14:56:13.370 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-08-21T14:56:13.372 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-08-21T14:56:13.406 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:13 +0000 (0:00:00.114) 0:01:23.918 ******* 2021-08-21T14:56:13.478 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* ok: [smithi165.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-08-21T14:56:13.502 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:13 +0000 (0:00:00.097) 0:01:24.016 ******* 2021-08-21T14:56:13.581 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-08-21T14:56:13.584 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "lttng-tools", "libtool-bin", "docker.io", "python3-nose", "python3-venv", "python3-virtualenv", "python3-configobj", "python3-gevent", "python3-numpy", "python3-matplotlib", "python3-setuptools"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_20.yml"} 2021-08-21T14:56:13.606 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:13 +0000 (0:00:00.103) 0:01:24.119 ******* 2021-08-21T14:56:13.685 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi165.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-08-21T14:56:13.710 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:13 +0000 (0:00:00.104) 0:01:24.223 ******* 2021-08-21T14:56:13.853 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-08-21T14:56:13.876 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:13 +0000 (0:00:00.166) 0:01:24.390 ******* 2021-08-21T14:56:14.051 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-08-21T14:56:14.053 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-08-21T14:56:14.079 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:14 +0000 (0:00:00.201) 0:01:24.591 ******* 2021-08-21T14:56:14.148 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-08-21T14:56:14.150 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:14.180 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:14 +0000 (0:00:00.102) 0:01:24.693 ******* 2021-08-21T14:56:14.249 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-08-21T14:56:14.250 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:14.274 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:14 +0000 (0:00:00.094) 0:01:24.788 ******* 2021-08-21T14:56:14.491 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-08-21T14:56:14.493 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-08-21T14:56:14.517 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:14 +0000 (0:00:00.242) 0:01:25.030 ******* 2021-08-21T14:56:14.983 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-08-21T14:56:14.985 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-08-21T14:56:15.007 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:15 +0000 (0:00:00.489) 0:01:25.520 ******* 2021-08-21T14:56:15.086 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi165.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi165"}, "changed": false} 2021-08-21T14:56:15.113 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:15 +0000 (0:00:00.106) 0:01:25.627 ******* 2021-08-21T14:56:15.957 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-08-21T14:56:15.959 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi165.front.sepia.ceph.com", "ansible_hostname": "smithi165", "ansible_nodename": "smithi165"}, "changed": true, "name": "smithi165"} 2021-08-21T14:56:15.981 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:15 +0000 (0:00:00.867) 0:01:26.494 ******* 2021-08-21T14:56:16.363 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-08-21T14:56:16.365 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-08-21T14:56:16.387 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:16 +0000 (0:00:00.405) 0:01:26.900 ******* 2021-08-21T14:56:16.769 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-08-21T14:56:16.771 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-08-21T14:56:16.795 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:16 +0000 (0:00:00.407) 0:01:27.308 ******* 2021-08-21T14:56:17.307 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-08-21T14:56:17.308 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-08-21T14:56:17.331 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:17 +0000 (0:00:00.536) 0:01:27.844 ******* 2021-08-21T14:56:17.402 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:17.434 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:17 +0000 (0:00:00.102) 0:01:27.947 ******* 2021-08-21T14:56:17.504 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-08-21T14:56:17.508 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:17.528 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:17 +0000 (0:00:00.094) 0:01:28.042 ******* 2021-08-21T14:56:17.607 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-08-21T14:56:17.609 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:17.634 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:17 +0000 (0:00:00.105) 0:01:28.147 ******* 2021-08-21T14:56:17.694 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-08-21T14:56:17.696 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:17.718 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:17 +0000 (0:00:00.085) 0:01:28.232 ******* 2021-08-21T14:56:17.787 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:17.812 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:17 +0000 (0:00:00.093) 0:01:28.326 ******* 2021-08-21T14:56:17.881 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:17.905 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:17 +0000 (0:00:00.092) 0:01:28.418 ******* 2021-08-21T14:56:17.974 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-08-21T14:56:17.975 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:17.998 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:17 +0000 (0:00:00.093) 0:01:28.511 ******* 2021-08-21T14:56:18.068 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:18.092 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:18 +0000 (0:00:00.094) 0:01:28.606 ******* 2021-08-21T14:56:18.178 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-08-21T14:56:18.180 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:18.202 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:18 +0000 (0:00:00.109) 0:01:28.715 ******* 2021-08-21T14:56:18.282 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* skipping: [smithi165.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-08-21T14:56:18.306 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:18 +0000 (0:00:00.103) 0:01:28.819 ******* 2021-08-21T14:56:18.386 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:18.411 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:18 +0000 (0:00:00.105) 0:01:28.924 ******* 2021-08-21T14:56:18.495 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-08-21T14:56:18.520 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:18 +0000 (0:00:00.108) 0:01:29.033 ******* 2021-08-21T14:56:18.602 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-08-21T14:56:18.625 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:18 +0000 (0:00:00.105) 0:01:29.139 ******* 2021-08-21T14:56:18.706 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-08-21T14:56:18.730 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:18 +0000 (0:00:00.104) 0:01:29.243 ******* 2021-08-21T14:56:18.799 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:18.827 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:18 +0000 (0:00:00.096) 0:01:29.340 ******* 2021-08-21T14:56:18.897 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi165.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-08-21T14:56:18.908 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-08-21T14:56:18.933 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:18 +0000 (0:00:00.106) 0:01:29.446 ******* 2021-08-21T14:56:19.002 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:19.025 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:19 +0000 (0:00:00.091) 0:01:29.538 ******* 2021-08-21T14:56:19.093 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-08-21T14:56:19.095 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:19.117 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:19 +0000 (0:00:00.092) 0:01:29.631 ******* 2021-08-21T14:56:19.186 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-08-21T14:56:19.188 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:19.210 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:19 +0000 (0:00:00.092) 0:01:29.724 ******* 2021-08-21T14:56:19.281 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:19.304 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:19 +0000 (0:00:00.094) 0:01:29.818 ******* 2021-08-21T14:56:19.373 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:19.398 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:19 +0000 (0:00:00.093) 0:01:29.911 ******* 2021-08-21T14:56:19.467 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-08-21T14:56:19.469 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:19.491 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:19 +0000 (0:00:00.092) 0:01:30.004 ******* 2021-08-21T14:56:19.559 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-08-21T14:56:19.562 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:19.583 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:19 +0000 (0:00:00.092) 0:01:30.096 ******* 2021-08-21T14:56:19.651 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-08-21T14:56:19.653 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:19.674 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:19 +0000 (0:00:00.091) 0:01:30.188 ******* 2021-08-21T14:56:19.744 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:19.767 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:19 +0000 (0:00:00.092) 0:01:30.280 ******* 2021-08-21T14:56:19.836 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:19.860 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:19 +0000 (0:00:00.093) 0:01:30.374 ******* 2021-08-21T14:56:19.930 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-08-21T14:56:19.932 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:19.962 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:19 +0000 (0:00:00.100) 0:01:30.475 ******* 2021-08-21T14:56:20.032 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-08-21T14:56:20.033 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:20.057 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:20 +0000 (0:00:00.095) 0:01:30.570 ******* 2021-08-21T14:56:20.126 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:20.150 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:20 +0000 (0:00:00.093) 0:01:30.664 ******* 2021-08-21T14:56:20.240 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-08-21T14:56:20.242 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:20.265 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:20 +0000 (0:00:00.114) 0:01:30.778 ******* 2021-08-21T14:56:20.336 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-08-21T14:56:20.337 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:20.360 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:20 +0000 (0:00:00.094) 0:01:30.873 ******* 2021-08-21T14:56:20.430 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-08-21T14:56:20.432 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:20.455 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:20 +0000 (0:00:00.095) 0:01:30.969 ******* 2021-08-21T14:56:20.525 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:20.550 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:20 +0000 (0:00:00.094) 0:01:31.064 ******* 2021-08-21T14:56:20.632 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:20.655 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:20 +0000 (0:00:00.105) 0:01:31.169 ******* 2021-08-21T14:56:20.735 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:20.770 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:20 +0000 (0:00:00.109) 0:01:31.278 ******* 2021-08-21T14:56:20.834 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-08-21T14:56:20.836 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:20.858 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:20 +0000 (0:00:00.093) 0:01:31.372 ******* 2021-08-21T14:56:21.074 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005296", "end": "2021-08-21 14:56:20.650767", "rc": 0, "start": "2021-08-21 14:56:20.645471", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:56:21.099 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:21 +0000 (0:00:00.240) 0:01:31.613 ******* 2021-08-21T14:56:21.167 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-08-21T14:56:21.193 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:21 +0000 (0:00:00.093) 0:01:31.707 ******* 2021-08-21T14:56:21.596 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-08-21T14:56:21.598 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-08-21T14:56:21.621 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:21 +0000 (0:00:00.426) 0:01:32.133 ******* 2021-08-21T14:56:21.715 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:21.744 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:21 +0000 (0:00:00.123) 0:01:32.257 ******* 2021-08-21T14:56:23.068 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* ok: [smithi165.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-08-21T14:56:24.053 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-08-21T14:56:24.077 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:24 +0000 (0:00:02.333) 0:01:34.591 ******* 2021-08-21T14:56:24.814 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-08-21T14:56:24.815 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:56:24.842 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:24 +0000 (0:00:00.764) 0:01:35.355 ******* 2021-08-21T14:56:24.912 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-08-21T14:56:24.936 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:24 +0000 (0:00:00.094) 0:01:35.450 ******* 2021-08-21T14:56:25.146 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-08-21T14:56:25.147 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.028541", "end": "2021-08-21 14:56:24.730157", "rc": 0, "start": "2021-08-21 14:56:24.701616", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:56:25.175 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:25 +0000 (0:00:00.238) 0:01:35.688 ******* 2021-08-21T14:56:27.374 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** ok: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:56:27.398 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:27 +0000 (0:00:02.223) 0:01:37.911 ******* 2021-08-21T14:56:35.032 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-08-21T14:56:35.034 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1\n libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0\n libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0\n libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0\n libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla\n libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0\n libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf\n qemu-system-data qemu-system-gui seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 168 not upgraded.\nAfter this operation, 62.7 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138108 files and directories currently installed.)\r 2021-08-21T14:56:35.036 INFO:teuthology.task.ansible.out:\nRemoving qemu-utils (1:4.2-3ubuntu6.17) ...\r\nRemoving qemu-system-x86 (1:4.2-3ubuntu6.17) ...\r\nRemoving qemu-system-common (1:4.2-3ubuntu6.17) ...\r\nRemoving qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...\r\nRemoving librbd1 (15.2.13-0ubuntu0.20.04.1) ...\r\nRemoving librados2 (15.2.13-0ubuntu0.20.04.1) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1", " libboost-iostreams1.71.0 libboost-thread1.71.0 libbrlapi0.7 libcacard0", " libcdparanoia0 libdv4 libgbm1 libgstreamer-plugins-base1.0-0", " libgstreamer-plugins-good1.0-0 libgudev-1.0-0 libiscsi7 liborc-0.4-0", " libpmem1 libshout3 libslirp0 libspice-server1 libtag1v5 libtag1v5-vanilla", " libusbredirparser1 libv4l-0 libv4lconvert0 libvirglrenderer1 libvisual-0.4-0", " libvte-2.91-0 libvte-2.91-common libwayland-server0 msr-tools ovmf", " qemu-system-data qemu-system-gui seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 168 not upgraded.", "After this operation, 62.7 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "( 2021-08-21T14:56:35.037 INFO:teuthology.task.ansible.out:Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138108 files and directories currently installed.)", "Removing qemu-utils (1:4.2-3ubuntu6.17) ...", "Removing qemu-system-x86 (1:4.2-3ubuntu6.17) ...", "Removing qemu-system-common (1:4.2-3ubuntu6.17) ...", "Removing qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...", "Removing librbd1 (15.2.13-0ubuntu0.20.04.1) ...", "Removing librados2 (15.2.13-0ubuntu0.20.04.1) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ..."]} 2021-08-21T14:56:35.056 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:35 +0000 (0:00:07.657) 0:01:45.569 ******* 2021-08-21T14:56:35.771 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-08-21T14:56:35.773 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:56:35.795 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:35 +0000 (0:00:00.738) 0:01:46.308 ******* 2021-08-21T14:56:35.864 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:35.891 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:35 +0000 (0:00:00.095) 0:01:46.404 ******* 2021-08-21T14:56:54.280 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-08-21T14:56:54.282 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.25.1-1ubuntu3.1).\nThe following additional packages will be installed:\n libpython3.8 libpython3.8-dev libpython3.8-minimal libpython3.8-stdlib\n librados2 librbd1 python3.8 python3.8-dev python3.8-minimal python3.8-venv\n qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n python3.8-doc binfmt-support samba vde2 debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 python3-venv python3.8-venv qemu-block-extra\n qemu-system-common qemu-system-x86 qemu-utils\nThe following packages will be upgraded:\n libpython3.8 libpython3.8-dev libpython3.8-minimal libpython3.8-stdlib\n python3.8 python3.8-dev python3.8-minimal\n7 upgraded, 8 newly installed, 0 to remove and 161 not upgraded.\nNeed to get 24.4 MB of archives.\nAfter this operation, 62.8 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-dev amd64 3.8.10-0ubuntu1~20.04 [510 kB]\nGet:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8 amd64 3.8.10-0ubuntu1~20.04 [387 kB]\nGet:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-dev amd64 3.8.10-0ubuntu1~20.04 [3943 kB]\nGet:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8 amd64 3.8.10-0ubuntu1~20.04 [1625 kB]\nGet:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-stdlib amd64 3.8.10-0ubuntu1~20.04 [1675 kB]\nGet:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-minimal amd64 3.8.10-0ubuntu1~20.04 2021-08-21T14:56:54.283 INFO:teuthology.task.ansible.out:[1898 kB]\nGet:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-minimal amd64 3.8.10-0ubuntu1~20.04 [717 kB]\nGet:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.13-0ubuntu0.20.04.1 [3212 kB]\nGet:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.13-0ubuntu0.20.04.1 [1623 kB]\nGet:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3.8-venv amd64 3.8.10-0ubuntu1~20.04 [5456 B]\nGet:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-venv amd64 3.8.2-0ubuntu2 [1228 B]\nGet:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.17 [54.4 kB]\nGet:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.17 [1055 kB]\nGet:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.17 [6738 kB]\nGet:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.17 [973 kB]\nFetched 24.4 MB in 2s (15.7 MB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 138036 files and directories currently installed.)\r\nPreparing to unpack .../00-python3.8-dev_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking python3.8-dev (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../01-python3.8_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking python3.8 (3.8.10-0ubuntu1~2 2021-08-21T14:56:54.285 INFO:teuthology.task.ansible.out:0.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../02-libpython3.8-dev_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../03-libpython3.8_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking libpython3.8:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../04-libpython3.8-stdlib_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../05-python3.8-minimal_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking python3.8-minimal (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nPreparing to unpack .../06-libpython3.8-minimal_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...\r\nSelecting previously unselected package librados2.\r\nPreparing to unpack .../07-librados2_15.2.13-0ubuntu0.20.04.1_amd64.deb ...\r\nUnpacking librados2 (15.2.13-0ubuntu0.20.04.1) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../08-librbd1_15.2.13-0ubuntu0.20.04.1_amd64.deb ...\r\nUnpacking librbd1 (15.2.13-0ubuntu0.20.04.1) ...\r\nSelecting previously unselected package python3.8-venv.\r\nPreparing to unpack .../09-python3.8-venv_3.8.10-0ubuntu1~20.04_amd64.deb ...\r\nUnpacking python3.8-venv (3.8.10-0ubuntu1~20.04) ...\r\nSelecting previously unselected package python3-venv.\r\nPreparing to unpack .../10-python3-venv_3.8.2-0ubuntu2_amd64.deb ...\r\nUnpacking python3-venv (3.8.2-0ubuntu2) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../11-qemu-block-extra_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../12-qemu-system-common_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-sys 2021-08-21T14:56:54.287 INFO:teuthology.task.ansible.out:tem-common (1:4.2-3ubuntu6.17) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../13-qemu-system-x86_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:4.2-3ubuntu6.17) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../14-qemu-utils_1%3a4.2-3ubuntu6.17_amd64.deb ...\r\nUnpacking qemu-utils (1:4.2-3ubuntu6.17) ...\r\nSetting up librados2 (15.2.13-0ubuntu0.20.04.1) ...\r\nSetting up libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up librbd1 (15.2.13-0ubuntu0.20.04.1) ...\r\nSetting up python3.8-minimal (3.8.10-0ubuntu1~20.04) ...\r\nSetting up libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up python3.8 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...\r\nSetting up qemu-system-common (1:4.2-3ubuntu6.17) ...\r\nSetting up qemu-system-x86 (1:4.2-3ubuntu6.17) ...\r\nSetting up libpython3.8:amd64 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up python3.8-venv (3.8.10-0ubuntu1~20.04) ...\r\nSetting up qemu-utils (1:4.2-3ubuntu6.17) ...\r\nSetting up python3-venv (3.8.2-0ubuntu2) ...\r\nSetting up libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04) ...\r\nSetting up python3.8-dev (3.8.10-0ubuntu1~20.04) ...\r\nProcessing triggers for libc-bin (2.31-0ubuntu9.2) ...\r\nProcessing triggers for man-db (2.9.1-1) ...\r\nProcessing triggers for mime-support (3.64ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.25.1-1ubuntu3.1).", "The following additional packages will be installed:", " libpython3.8 libpython3.8-dev libpython3.8-minimal libpython3.8-stdlib", " librados2 librbd1 python3.8 python3.8-dev python3.8-minimal python3.8-venv", " qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " python3.8-doc binfmt-support samba vde2 debootstrap", "The following NEW packages will be installed:" 2021-08-21T14:56:54.288 INFO:teuthology.task.ansible.out:, " librados2 librbd1 python3-venv python3.8-venv qemu-block-extra", " qemu-system-common qemu-system-x86 qemu-utils", "The following packages will be upgraded:", " libpython3.8 libpython3. 2021-08-21T14:56:54.289 INFO:teuthology.task.ansible.out:8-dev libpython3.8-minimal libpython3.8-stdlib", " python3.8 python3.8-dev python3.8-minimal", "7 upgraded, 8 newly installed, 0 to remove and 161 not upgraded.", "Need to get 24.4 MB of archives.", "After this operation, 62.8 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-dev amd64 3.8.10-0ubuntu1~20.04 [510 kB]", "Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8 amd64 3.8.10-0ubuntu1~20.04 [387 kB]", "Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-dev amd64 3.8.10-0ubuntu1~20.04 [3943 kB]", "Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8 amd64 3.8.10-0ubuntu1~20.04 [1625 kB]", "Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-stdlib amd64 3.8.10-0ubuntu1~20.04 [1675 kB]", "Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3.8-minimal amd64 3.8.10-0ubuntu1~20.04 [1898 kB]", "Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpython3.8-minimal amd64 3.8.10-0ubuntu1~20.04 [717 kB]", "Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.13-0ubuntu0.20.04.1 [3212 kB]", "Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.13-0ubuntu0.20.04.1 [1623 kB]", "Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3.8-venv amd64 3.8.10-0ubuntu1~20.04 [5456 B]", "Get:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-venv amd64 3.8.2-0ubuntu2 [1228 B]", "Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.17 [54.4 kB]", "Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.1 2021-08-21T14:56:54.290 INFO:teuthology.task.ansible.out:7 [1055 kB]", "Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.17 [6738 kB]", "Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.17 [973 kB]", "Fetched 24.4 MB in 2s (15.7 MB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 138036 files and directories currently installed.)", "Preparing to unpack .../00-python3.8-dev_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking python3.8-dev (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../01-python3.8_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking python3.8 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../02-libpython3.8-dev_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../03-libpython3.8_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking libpython3.8:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../04-libpython3.8-stdlib_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../05-python3.8-minimal_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking python3.8-minimal (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Preparing to unpack .../06-libpython3.8-minimal_3.8.10 2021-08-21T14:56:54.292 INFO:teuthology.task.ansible.out:-0ubuntu1~20.04_amd64.deb ...", "Unpacking libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04) over (3.8.5-1~20.04.2) ...", "Selecting previously unselected package librados2.", "Preparing to unpack .../07-librados2_15.2.13-0ubuntu0.20.04.1_amd64.deb ...", "Unpacking librados2 (15.2.13-0ubuntu0.20.04.1) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../08-librbd1_15.2.13-0ubuntu0.20.04.1_amd64.deb ...", "Unpacking librbd1 (15.2.13-0ubuntu0.20.04.1) ...", "Selecting previously unselected package python3.8-venv.", "Preparing to unpack .../09-python3.8-venv_3.8.10-0ubuntu1~20.04_amd64.deb ...", "Unpacking python3.8-venv (3.8.10-0ubuntu1~20.04) ...", "Selecting previously unselected package python3-venv.", "Preparing to unpack .../10-python3-venv_3.8.2-0ubuntu2_amd64.deb ...", "Unpacking python3-venv (3.8.2-0ubuntu2) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../11-qemu-block-extra_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../12-qemu-system-common_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-system-common (1:4.2-3ubuntu6.17) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../13-qemu-system-x86_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-system-x86 (1:4.2-3ubuntu6.17) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../14-qemu-utils_1%3a4.2-3ubuntu6.17_amd64.deb ...", "Unpacking qemu-utils (1:4.2-3ubuntu6.17) ...", "Setting up librados2 (15.2.13-0ubuntu0.20.04.1) ...", "Setting up libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04) ...", "Setting up librbd1 (15.2.13-0ubuntu0.20.04.1) ...", "Setting up python3.8-minimal (3.8.10-0ubuntu1~20.04) ...", "Setting up libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04) ...", "Setting up python3.8 (3.8.10-0ubuntu1~20.04) ...", 2021-08-21T14:56:54.294 INFO:teuthology.task.ansible.out: "Setting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.17) ...", "Setting up qemu-system-common (1:4.2-3ubuntu6.17) ...", "Setting up qemu-system-x86 (1:4.2-3ubuntu6.17) ...", "Setting up libpython3.8:amd64 (3.8.10-0ubuntu1~20.04) ...", "Setting up python3.8-venv (3.8.10-0ubuntu1~20.04) ...", "Setting up qemu-utils (1:4.2-3ubuntu6.17) ...", "Setting up python3-venv (3.8.2-0ubuntu2) ...", "Setting up libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04) ...", "Setting up python3.8-dev (3.8.10-0ubuntu1~20.04) ...", "Processing triggers for libc-bin (2.31-0ubuntu9.2) ...", "Processing triggers for man-db (2.9.1-1) ...", "Processing triggers for mime-support (3.64ubuntu1) ..."]} 2021-08-21T14:56:54.314 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:54 +0000 (0:00:18.423) 0:02:04.828 ******* 2021-08-21T14:56:55.032 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* ok: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:56:55.055 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:55 +0000 (0:00:00.741) 0:02:05.569 ******* 2021-08-21T14:56:55.762 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-08-21T14:56:55.763 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:56:55.790 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:55 +0000 (0:00:00.734) 0:02:06.303 ******* 2021-08-21T14:56:55.981 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-08-21T14:56:56.007 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:56 +0000 (0:00:00.215) 0:02:06.519 ******* 2021-08-21T14:56:56.076 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:56.099 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:56 +0000 (0:00:00.092) 0:02:06.612 ******* 2021-08-21T14:56:56.283 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-08-21T14:56:56.309 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:56 +0000 (0:00:00.209) 0:02:06.822 ******* 2021-08-21T14:56:56.752 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-08-21T14:56:56.771 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:56 +0000 (0:00:00.462) 0:02:07.285 ******* 2021-08-21T14:56:56.983 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ ok: [smithi165.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-08-21T14:56:57.006 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:57 +0000 (0:00:00.234) 0:02:07.519 ******* 2021-08-21T14:56:57.191 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-08-21T14:56:57.194 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-08-21T14:56:57.216 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:57 +0000 (0:00:00.209) 0:02:07.729 ******* 2021-08-21T14:56:57.542 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-08-21T14:56:57.543 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Sat 2021-08-21 14:54:39 UTC", "ActiveEnterTimestampMonotonic": "14630788", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service systemd-journald.socket basic.target remote-fs.target nss-lookup.target -.mount sysinit.target system.slice network.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-08-21 14:54:38 UTC", "AssertTimestampMonotonic": "13392305", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-08-21 14:54:38 UTC", "ConditionTimestampMonotonic": "13392305", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup" 2021-08-21T14:56:57.545 INFO:teuthology.task.ansible.out:: "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "Documentation": "https://httpd.apache.org/docs/2.4/", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "986", "ExecMainStartTimestamp": "Sat 2021-08-21 14:54:39 UTC", "ExecMainStartTimestampMonotonic": "14630763", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopEx": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709 2021-08-21T14:56:57.546 INFO:teuthology.task.ansible.out:551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-08-21 14:54:38 UTC", "InactiveExitTimestampMonotonic": "13393202", "InvocationID": "304f5cf2b8244ec4a884af1d525c1679", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127777", "LimitNPROCSoft": "127777", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "986", "MemoryAccounting": "yes", "MemoryCurrent": "9986048", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy 2021-08-21T14:56:57.548 INFO:teuthology.task.ansible.out:": "n/a", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-08-21 14:54:39 UTC", "StateChangeTimestampMonotonic": "14630788", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "38333", "TimeoutAbortUSec": " 2021-08-21T14:56:57.550 INFO:teuthology.task.ansible.out:1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-08-21T14:56:57.568 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:57 +0000 (0:00:00.352) 0:02:08.082 ******* 2021-08-21T14:56:57.638 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-08-21T14:56:57.639 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:57.674 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:57 +0000 (0:00:00.105) 0:02:08.187 ******* 2021-08-21T14:56:57.760 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-08-21T14:56:57.762 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:57.786 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:57 +0000 (0:00:00.112) 0:02:08.300 ******* 2021-08-21T14:56:57.858 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-08-21T14:56:57.860 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:57.882 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:57 +0000 (0:00:00.095) 0:02:08.396 ******* 2021-08-21T14:56:57.971 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-08-21T14:56:57.973 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:57.996 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:57 +0000 (0:00:00.113) 0:02:08.509 ******* 2021-08-21T14:56:58.064 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-08-21T14:56:58.065 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:58.090 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:58 +0000 (0:00:00.093) 0:02:08.602 ******* 2021-08-21T14:56:58.169 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-08-21T14:56:58.170 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:58.193 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:58 +0000 (0:00:00.104) 0:02:08.707 ******* 2021-08-21T14:56:58.263 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:58.288 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:58 +0000 (0:00:00.093) 0:02:08.801 ******* 2021-08-21T14:56:58.358 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-08-21T14:56:58.360 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:58.383 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:58 +0000 (0:00:00.095) 0:02:08.896 ******* 2021-08-21T14:56:58.462 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:58.486 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:58 +0000 (0:00:00.103) 0:02:09.000 ******* 2021-08-21T14:56:58.556 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:58.580 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:58 +0000 (0:00:00.093) 0:02:09.093 ******* 2021-08-21T14:56:58.648 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:58.671 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:58 +0000 (0:00:00.091) 0:02:09.185 ******* 2021-08-21T14:56:58.741 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-08-21T14:56:58.743 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:58.771 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:58 +0000 (0:00:00.099) 0:02:09.285 ******* 2021-08-21T14:56:58.851 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-08-21T14:56:58.853 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:58.878 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:58 +0000 (0:00:00.106) 0:02:09.391 ******* 2021-08-21T14:56:58.961 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-08-21T14:56:58.962 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:58.995 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:58 +0000 (0:00:00.117) 0:02:09.508 ******* 2021-08-21T14:56:59.064 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-08-21T14:56:59.066 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:59.088 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:59 +0000 (0:00:00.093) 0:02:09.602 ******* 2021-08-21T14:56:59.158 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-08-21T14:56:59.160 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:59.187 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:59 +0000 (0:00:00.098) 0:02:09.700 ******* 2021-08-21T14:56:59.258 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-08-21T14:56:59.260 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:59.282 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:59 +0000 (0:00:00.094) 0:02:09.795 ******* 2021-08-21T14:56:59.350 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-08-21T14:56:59.352 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:59.380 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:59 +0000 (0:00:00.097) 0:02:09.893 ******* 2021-08-21T14:56:59.451 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:59.476 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:59 +0000 (0:00:00.095) 0:02:09.989 ******* 2021-08-21T14:56:59.544 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-08-21T14:56:59.547 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:59.568 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:59 +0000 (0:00:00.092) 0:02:10.082 ******* 2021-08-21T14:56:59.640 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:59.669 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:59 +0000 (0:00:00.097) 0:02:10.179 ******* 2021-08-21T14:56:59.737 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-08-21T14:56:59.740 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:59.762 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:59 +0000 (0:00:00.096) 0:02:10.275 ******* 2021-08-21T14:56:59.831 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-08-21T14:56:59.833 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:59.856 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:59 +0000 (0:00:00.094) 0:02:10.370 ******* 2021-08-21T14:56:59.927 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:56:59.954 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:56:59 +0000 (0:00:00.097) 0:02:10.467 ******* 2021-08-21T14:57:00.024 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:00.048 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:00 +0000 (0:00:00.094) 0:02:10.562 ******* 2021-08-21T14:57:00.128 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:00.166 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:00 +0000 (0:00:00.117) 0:02:10.679 ******* 2021-08-21T14:57:00.352 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-08-21T14:57:00.376 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:00 +0000 (0:00:00.210) 0:02:10.889 ******* 2021-08-21T14:57:00.779 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-08-21T14:57:00.814 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:00 +0000 (0:00:00.437) 0:02:11.327 ******* 2021-08-21T14:57:00.894 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:00.931 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:00 +0000 (0:00:00.112) 0:02:11.440 ******* 2021-08-21T14:57:01.356 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-08-21T14:57:01.358 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-08-21T14:57:01.380 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:01 +0000 (0:00:00.454) 0:02:11.894 ******* 2021-08-21T14:57:01.807 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-08-21T14:57:01.838 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:01 +0000 (0:00:00.457) 0:02:12.352 ******* 2021-08-21T14:57:02.273 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-08-21T14:57:02.274 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-08-21T14:57:02.305 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:02 +0000 (0:00:00.467) 0:02:12.819 ******* 2021-08-21T14:57:02.492 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-08-21T14:57:02.494 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-08-21T14:57:02.527 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:02 +0000 (0:00:00.221) 0:02:13.040 ******* 2021-08-21T14:57:02.598 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-08-21T14:57:02.600 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:02.629 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:02 +0000 (0:00:00.097) 0:02:13.138 ******* 2021-08-21T14:57:02.685 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:02.708 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:02 +0000 (0:00:00.083) 0:02:13.221 ******* 2021-08-21T14:57:02.779 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:02.803 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:02 +0000 (0:00:00.095) 0:02:13.317 ******* 2021-08-21T14:57:02.875 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:02.899 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:02 +0000 (0:00:00.095) 0:02:13.412 ******* 2021-08-21T14:57:02.979 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-08-21T14:57:02.981 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.003 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:03 +0000 (0:00:00.104) 0:02:13.516 ******* 2021-08-21T14:57:03.079 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi165.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-08-21T14:57:03.103 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:03 +0000 (0:00:00.099) 0:02:13.616 ******* 2021-08-21T14:57:03.214 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-08-21T14:57:03.216 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.229 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.231 INFO:teuthology.task.ansible.out: 2021-08-21T14:57:03.252 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.263 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.274 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.285 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-08-21T14:57:03.296 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.307 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.309 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.318 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.340 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.342 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.373 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:03 +0000 (0:00:00.269) 0:02:13.886 ******* 2021-08-21T14:57:03.645 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-08-21T14:57:03.647 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.008485", "end": "2021-08-21 14:57:03.619572", "rc": 0, "start": "2021-08-21 14:57:03.611087", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/bin/teuthology-lock"]} 2021-08-21T14:57:03.679 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:03 +0000 (0:00:00.306) 0:02:14.192 ******* 2021-08-21T14:57:03.749 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-08-21T14:57:03.751 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.779 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:03 +0000 (0:00:00.100) 0:02:14.293 ******* 2021-08-21T14:57:03.869 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-08-21T14:57:03.871 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.893 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:03 +0000 (0:00:00.113) 0:02:14.407 ******* 2021-08-21T14:57:03.962 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-08-21T14:57:03.964 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:03.990 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:03 +0000 (0:00:00.096) 0:02:14.504 ******* 2021-08-21T14:57:04.685 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-08-21T14:57:04.687 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:57:04.708 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:04 +0000 (0:00:00.717) 0:02:15.222 ******* 2021-08-21T14:57:04.778 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:04.800 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:04 +0000 (0:00:00.092) 0:02:15.314 ******* 2021-08-21T14:57:04.882 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ ok: [smithi165.front.sepia.ceph.com] => (item={'block_used': 13866673, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226228435, 'size_available': 926631669760, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150395, 'block_size': 4096, 'inode_available': 60904581}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226228435, "block_size": 4096, "block_total": 240095108, "block_used": 13866673, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904581, "inode_total": 61054976, "inode_used": 150395, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926631669760, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}} 2021-08-21T14:57:04.907 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:04 +0000 (0:00:00.106) 0:02:15.420 ******* 2021-08-21T14:57:05.101 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-08-21T14:57:05.103 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009142", "end": "2021-08-21 14:57:04.683628", "rc": 0, "start": "2021-08-21 14:57:04.674486", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:57:05.141 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:05 +0000 (0:00:00.234) 0:02:15.654 ******* 2021-08-21T14:57:05.224 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-08-21T14:57:05.226 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'block_used': 13866673, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226228435, 'size_available': 926631669760, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150395, 'block_size': 4096, 'inode_available': 60904581}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226228435, "block_size": 4096, "block_total": 240095108, "block_used": 13866673, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904581, "inode_total": 61054976, "inode_used": 150395, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926631669760, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:05.249 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:05 +0000 (0:00:00.107) 0:02:15.762 ******* 2021-08-21T14:57:05.318 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-08-21T14:57:05.340 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:05 +0000 (0:00:00.091) 0:02:15.853 ******* 2021-08-21T14:57:05.433 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:07.597 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7A2jIWTHlMa9Nva646S24DAjZU02jIAL2'], 'uuids': ['755fc437-018f-49e2-9ac1-54a5f2ef19b1']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:02.024415", "end": "2021-08-21 14:57:07.170522", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7A2jIWTHlMa9Nva646S24DAjZU02jIAL2"], "labels": [], "masters": [], "uuids": ["755fc437-018f-49e2-9ac1-54a5f2ef19b1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:57:05.146107", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-08-21T14:57:07.610 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:08.874 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7kcqxYmNAxwUPUNay0heeNixphXQAo4rQ'], 'uuids': ['bbd5a896-3d38-4270-97a1-e13d94d10995']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.137155", "end": "2021-08-21 14:57:08.453948", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7kcqxYmNAxwUPUNay0heeNixphXQAo4rQ"], "labels": [], "masters": [], "uuids": ["bbd5a896-3d38-4270-97a1-e13d94d10995"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:57:07.316793", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-08-21T14:57:08.885 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09', 'wwn-0x5000c500929239ac'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09-part1', 'wwn-0x5000c500929239ac-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500929239ac', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XG09', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XG09", "wwn-0x5000c500929239ac"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XG09-part1", "wwn-0x5000c500929239ac-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XG09", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500929239ac"}}, 2021-08-21T14:57:08.887 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2021-08-21T14:57:10.183 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-oRFRKa-pp1D-I4g3-8OSj-nrEC-k7wY-9se4GS', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300CE400BGN', 'nvme-nvme.8086-43564654363233333030434534303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.169213", "end": "2021-08-21 14:57:09.760415", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-oRFRKa-pp1D-I4g3-8OSj-nrEC-k7wY-9se4GS", "nvme-INTEL_SSDPEDMD400G4_CVFT623300CE400BGN", "nvme-nvme.8086-43564654363233333030434534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:57:08.591202", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nGPT data structures d 2021-08-21T14:57:10.185 INFO:teuthology.task.ansible.out:estroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-08-21T14:57:10.192 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:10.203 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:11.485 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7crsB64tRrekCoHhBeCqCDroAOmIhXx8e'], 'uuids': ['17e22f6b-a6c1-44a4-a871-f4ff116767b6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.155993", "end": "2021-08-21 14:57:11.064404", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7crsB64tRrekCoHhBeCqCDroAOmIhXx8e"], "labels": [], "masters": [], "uuids": ["17e22f6b-a6c1-44a4-a871-f4ff116767b6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:57:09.908411", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-08-21T14:57:11.497 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:11.509 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:12.777 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7b9gTKXBpypke2qRDqa5s8DaoP1IgoFn3'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.136397", "end": "2021-08-21 14:57:12.350250", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7b9gTKXBpypke2qRDqa5s8DaoP1IgoFn3"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:57:11.213853", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-08-21T14:57:12.789 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:14.245 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp77dekcvz60ZFUmsqdfS8ipwT2W3lrTXgU'], 'uuids': ['b9ada875-4d81-45ae-a381-e4f20c360cde']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.329761", "end": "2021-08-21 14:57:13.822835", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp77dekcvz60ZFUmsqdfS8ipwT2W3lrTXgU"], "labels": [], "masters": [], "uuids": ["b9ada875-4d81-45ae-a381-e4f20c360cde"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:57:12.493074", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries in memory.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries in memory.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-08-21T14:57:14.290 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:14 +0000 (0:00:08.950) 0:02:24.804 ******* 2021-08-21T14:57:14.371 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-08-21T14:57:14.373 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:14.693 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7A2jIWTHlMa9Nva646S24DAjZU02jIAL2'], 'uuids': ['755fc437-018f-49e2-9ac1-54a5f2ef19b1']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.165352", "end": "2021-08-21 14:57:14.245755", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7A2jIWTHlMa9Nva646S24DAjZU02jIAL2"], "labels": [], "masters": [], "uuids": ["755fc437-018f-49e2-9ac1-54a5f2ef19b1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:57:14.080403", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.160561 s, 718 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.160561 s, 718 MB/s"], "stdout": "", "stdout_lines": []} 2021-08-21T14:57:14.695 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:14.696 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:14.989 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7kcqxYmNAxwUPUNay0heeNixphXQAo4rQ'], 'uuids': ['bbd5a896-3d38-4270-97a1-e13d94d10995']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.164235", "end": "2021-08-21 14:57:14.568623", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7kcqxYmNAxwUPUNay0heeNixphXQAo4rQ"], "labels": [], "masters": [], "uuids": ["bbd5a896-3d38-4270-97a1-e13d94d10995"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:57:14.404388", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.159365 s, 724 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.159365 s, 724 MB/s"], "stdout": "", "stdout_lines": []} 2021-08-21T14:57:15.001 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09', 'wwn-0x5000c500929239ac'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09-part1', 'wwn-0x5000c500929239ac-part1'], 'uuids': ['765c9454-acf9-4ab1-8c0e-444ef075a2c6']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500929239ac', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XG09', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XG09", "wwn-0x5000c500929239ac"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XG09-part1", "wwn-0x5000c500929239ac-part1"], "labels": [], "masters": [], "uuids": ["765c9454-acf9-4ab1-8c0e-444ef075a2c6"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XG09", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500929239ac"}}, 2021-08-21T14:57:15.003 INFO:teuthology.task.ansible.out: "skip_reason": "Conditional result was False"} 2021-08-21T14:57:15.180 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-oRFRKa-pp1D-I4g3-8OSj-nrEC-k7wY-9se4GS', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300CE400BGN', 'nvme-nvme.8086-43564654363233333030434534303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.058214", "end": "2021-08-21 14:57:14.764470", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-oRFRKa-pp1D-I4g3-8OSj-nrEC-k7wY-9se4GS", "nvme-INTEL_SSDPEDMD400G4_CVFT623300CE400BGN", "nvme-nvme.8086-43564654363233333030434534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:57:14.706256", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0533706 s, 2.2 GB/s", "std 2021-08-21T14:57:15.182 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0533706 s, 2.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-08-21T14:57:15.192 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:15.207 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:15.514 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7crsB64tRrekCoHhBeCqCDroAOmIhXx8e'], 'uuids': ['17e22f6b-a6c1-44a4-a871-f4ff116767b6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.171143", "end": "2021-08-21 14:57:15.083975", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7crsB64tRrekCoHhBeCqCDroAOmIhXx8e"], "labels": [], "masters": [], "uuids": ["17e22f6b-a6c1-44a4-a871-f4ff116767b6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:57:14.912832", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.16618 s, 694 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.16618 s, 694 MB/s"], "stdout": "", "stdout_lines": []} 2021-08-21T14:57:15.517 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:15.527 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:15.823 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7b9gTKXBpypke2qRDqa5s8DaoP1IgoFn3'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.165112", "end": "2021-08-21 14:57:15.398873", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp7b9gTKXBpypke2qRDqa5s8DaoP1IgoFn3"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:57:15.233761", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.160262 s, 720 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.160262 s, 720 MB/s"], "stdout": "", "stdout_lines": []} 2021-08-21T14:57:15.835 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'mq-deadline', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:16.123 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp77dekcvz60ZFUmsqdfS8ipwT2W3lrTXgU'], 'uuids': ['b9ada875-4d81-45ae-a381-e4f20c360cde']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.163636", "end": "2021-08-21 14:57:15.705346", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-gkeYCnNfrv3ZY5TkSUl1v9Ijrygr4Jp77dekcvz60ZFUmsqdfS8ipwT2W3lrTXgU"], "labels": [], "masters": [], "uuids": ["b9ada875-4d81-45ae-a381-e4f20c360cde"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:57:15.541710", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.158781 s, 726 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.158781 s, 726 MB/s"], "stdout": "", "stdout_lines": []} 2021-08-21T14:57:16.168 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:16 +0000 (0:00:01.878) 0:02:26.682 ******* 2021-08-21T14:57:16.500 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-08-21T14:57:16.502 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.138171", "end": "2021-08-21 14:57:16.024440", "rc": 0, "start": "2021-08-21 14:57:15.886269", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:57:16.525 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:16 +0000 (0:00:00.356) 0:02:27.038 ******* 2021-08-21T14:57:16.774 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-08-21T14:57:16.776 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.099429", "end": "2021-08-21 14:57:16.349272", "rc": 0, "start": "2021-08-21 14:57:16.249843", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:57:16.799 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:16 +0000 (0:00:00.273) 0:02:27.312 ******* 2021-08-21T14:57:16.868 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-08-21T14:57:16.926 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:16 +0000 (0:00:00.127) 0:02:27.439 ******* 2021-08-21T14:57:16.953 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:16.979 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:16 +0000 (0:00:00.053) 0:02:27.492 ******* 2021-08-21T14:57:17.007 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:17.033 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:17 +0000 (0:00:00.053) 0:02:27.546 ******* 2021-08-21T14:57:17.060 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-08-21T14:57:17.064 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:17.086 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:17 +0000 (0:00:00.053) 0:02:27.600 ******* 2021-08-21T14:57:17.113 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:17.140 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:17 +0000 (0:00:00.053) 0:02:27.653 ******* 2021-08-21T14:57:17.283 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi165.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-08-21T14:57:17.309 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:17 +0000 (0:00:00.168) 0:02:27.821 ******* 2021-08-21T14:57:17.356 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-08-21T14:57:17.358 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'block_used': 13866673, 'uuid': '765c9454-acf9-4ab1-8c0e-444ef075a2c6', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226228435, 'size_available': 926631669760, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro', 'device': '/dev/sda1', 'inode_used': 150395, 'block_size': 4096, 'inode_available': 60904581}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226228435, "block_size": 4096, "block_total": 240095108, "block_used": 13866673, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60904581, "inode_total": 61054976, "inode_used": 150395, "mount": "/", "options": "rw,relatime,errors=remount-ro", "size_available": 926631669760, "size_total": 983429562368, "uuid": "765c9454-acf9-4ab1-8c0e-444ef075a2c6"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:17.390 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:17 +0000 (0:00:00.081) 0:02:27.903 ******* 2021-08-21T14:57:17.440 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:17.464 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:17 +0000 (0:00:00.074) 0:02:27.978 ******* 2021-08-21T14:57:17.493 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-08-21T14:57:17.495 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:17.518 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:17 +0000 (0:00:00.052) 0:02:28.031 ******* 2021-08-21T14:57:17.556 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:17.591 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:17 +0000 (0:00:00.074) 0:02:28.105 ******* 2021-08-21T14:57:17.619 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:17.643 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:17 +0000 (0:00:00.052) 0:02:28.157 ******* 2021-08-21T14:57:17.681 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-08-21T14:57:17.683 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:17.707 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:17 +0000 (0:00:00.061) 0:02:28.219 ******* 2021-08-21T14:57:17.748 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-08-21T14:57:17.750 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:17.784 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:17 +0000 (0:00:00.076) 0:02:28.295 ******* 2021-08-21T14:57:18.651 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-08-21T14:57:18.677 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:18 +0000 (0:00:00.894) 0:02:29.190 ******* 2021-08-21T14:57:19.331 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-08-21T14:57:19.333 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-08-21T14:57:19.722 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-08-21T14:57:20.100 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-08-21T14:57:20.475 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-08-21T14:57:20.864 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-08-21T14:57:20.888 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:20 +0000 (0:00:02.211) 0:02:31.402 ******* 2021-08-21T14:57:21.030 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-08-21T14:57:21.032 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-08-21T14:57:21.055 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:21 +0000 (0:00:00.167) 0:02:31.569 ******* 2021-08-21T14:57:21.213 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-08-21T14:57:21.342 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-08-21T14:57:21.459 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-08-21T14:57:21.576 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-08-21T14:57:21.587 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:21.617 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:21 +0000 (0:00:00.557) 0:02:32.126 ******* 2021-08-21T14:57:21.640 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:21.663 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:21 +0000 (0:00:00.050) 0:02:32.176 ******* 2021-08-21T14:57:21.690 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:21.715 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:21 +0000 (0:00:00.051) 0:02:32.228 ******* 2021-08-21T14:57:21.858 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-08-21T14:57:21.881 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:21 +0000 (0:00:00.166) 0:02:32.394 ******* 2021-08-21T14:57:21.921 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-08-21T14:57:21.923 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-08-21T14:57:21.947 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:21 +0000 (0:00:00.066) 0:02:32.461 ******* 2021-08-21T14:57:22.452 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi165.front.sepia.ceph.com] => {"changed": true} 2021-08-21T14:57:22.485 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:22 +0000 (0:00:00.537) 0:02:32.998 ******* 2021-08-21T14:57:22.968 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-08-21T14:57:22.991 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:22 +0000 (0:00:00.506) 0:02:33.504 ******* 2021-08-21T14:57:23.018 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-08-21T14:57:23.019 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:23.043 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:23 +0000 (0:00:00.051) 0:02:33.556 ******* 2021-08-21T14:57:23.070 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:23.099 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:23 +0000 (0:00:00.056) 0:02:33.612 ******* 2021-08-21T14:57:23.129 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:23.153 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:23 +0000 (0:00:00.053) 0:02:33.666 ******* 2021-08-21T14:57:23.180 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-08-21T14:57:23.181 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:23.203 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:23 +0000 (0:00:00.050) 0:02:33.717 ******* 2021-08-21T14:57:23.361 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-08-21T14:57:23.363 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-08-21T14:57:23.389 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:23 +0000 (0:00:00.185) 0:02:33.902 ******* 2021-08-21T14:57:23.552 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-08-21T14:57:23.555 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-08-21T14:57:23.580 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:23 +0000 (0:00:00.191) 0:02:34.093 ******* 2021-08-21T14:57:23.927 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-08-21T14:57:23.932 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-08-21T14:57:23.962 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:23 +0000 (0:00:00.377) 0:02:34.471 ******* 2021-08-21T14:57:24.298 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-08-21T14:57:24.299 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-08-21T14:57:24.323 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:24 +0000 (0:00:00.365) 0:02:34.836 ******* 2021-08-21T14:57:24.990 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-08-21T14:57:24.994 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:57:25.014 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:25 +0000 (0:00:00.690) 0:02:35.527 ******* 2021-08-21T14:57:25.262 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-08-21T14:57:25.264 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.105090", "end": "2021-08-21 14:57:24.845583", "rc": 0, "start": "2021-08-21 14:57:24.740493", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.30.0/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.30.0/Amazon/S3.pm"]} 2021-08-21T14:57:25.287 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:25 +0000 (0:00:00.272) 0:02:35.800 ******* 2021-08-21T14:57:25.325 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-08-21T14:57:25.326 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:25.350 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:25 +0000 (0:00:00.061) 0:02:35.862 ******* 2021-08-21T14:57:25.375 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-08-21T14:57:25.377 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:25.402 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:25 +0000 (0:00:00.053) 0:02:35.915 ******* 2021-08-21T14:57:26.065 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-08-21T14:57:26.067 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:57:26.089 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:26 +0000 (0:00:00.687) 0:02:36.603 ******* 2021-08-21T14:57:26.117 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:26.140 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:26 +0000 (0:00:00.050) 0:02:36.654 ******* 2021-08-21T14:57:26.484 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-08-21T14:57:26.486 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-08-21T14:57:26.509 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:26 +0000 (0:00:00.368) 0:02:37.022 ******* 2021-08-21T14:57:26.536 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:26.560 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:26 +0000 (0:00:00.051) 0:02:37.073 ******* 2021-08-21T14:57:26.772 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-08-21T14:57:26.773 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-08-21 14:54:38 UTC", "ActiveEnterTimestampMonotonic": "13553559", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target systemd-tmpfiles-setup.service sysinit.target network.target systemd-journald.socket system.slice -.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-08-21 14:54:38 UTC", "AssertTimestampMonotonic": "13336266", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-08-21 14:54:38 UTC", "ConditionTimestampMonotonic": "13336265", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": 2021-08-21T14:57:26.774 INFO:teuthology.task.ansible.out: "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documentation": "man:ntpd(8)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "829", "ExecMainStartTimestamp": "Sat 2021-08-21 14:54:38 UTC", "ExecMainStartTimestampMonotonic": "13553547", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-08-21 14:54:38 UTC", "InactiveExitTimestampMonotonic": "13337284", "InvocationID": "9df04b6d0b3b4745bf562a15e912d667", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "Lim 2021-08-21T14:57:26.776 INFO:teuthology.task.ansible.out:itCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "524288", "LimitNOFILESoft": "1024", "LimitNPROC": "127777", "LimitNPROCSoft": "127777", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "829", "MemoryAccounting": "yes", "MemoryCurrent": "3686400", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMPolicy": "stop", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "Requi 2021-08-21T14:57:26.779 INFO:teuthology.task.ansible.out:resMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartKillSignal": "15", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-08-21 14:54:38 UTC", "StateChangeTimestampMonotonic": "13553559", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "38333", "TimeoutAbortUSec": "1min 30s", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-08-21T14:57:26.805 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:26 +0000 (0:00:00.240) 0:02:37.314 ******* 2021-08-21T14:57:26.830 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-08-21T14:57:26.832 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-08-21T14:57:26.853 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:26 +0000 (0:00:00.052) 0:02:37.367 ******* 2021-08-21T14:57:26.892 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi165.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-08-21T14:57:26.917 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:26 +0000 (0:00:00.063) 0:02:37.430 ******* 2021-08-21T14:57:26.944 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-08-21T14:57:26.946 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:26.977 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:26 +0000 (0:00:00.060) 0:02:37.491 ******* 2021-08-21T14:57:28.003 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* ok: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:57:28.025 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:28 +0000 (0:00:01.048) 0:02:38.539 ******* 2021-08-21T14:57:28.053 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-08-21T14:57:28.055 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:28.078 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:28 +0000 (0:00:00.052) 0:02:38.591 ******* 2021-08-21T14:57:28.265 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-08-21T14:57:28.266 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-08-21T14:57:28.289 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:28 +0000 (0:00:00.210) 0:02:38.802 ******* 2021-08-21T14:57:29.005 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-08-21T14:57:29.029 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:29 +0000 (0:00:00.740) 0:02:39.542 ******* 2021-08-21T14:57:31.250 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)\nRequirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.8/dist-packages (1.2.0)", "Requirement already satisfied: execnet in /usr/local/lib/python3.8/dist-packages (from remoto>=0.0.35) (1.8.0)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.8/dist-packages (from execnet->remoto>=0.0.35) (1.5)"], "version": null, "virtualenv": null} 2021-08-21T14:57:31.274 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:31 +0000 (0:00:02.245) 0:02:41.787 ******* 2021-08-21T14:57:31.797 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-08-21T14:57:31.799 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false} 2021-08-21T14:57:31.821 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:31 +0000 (0:00:00.546) 0:02:42.334 ******* 2021-08-21T14:57:31.860 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi165.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-08-21T14:57:31.884 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:31 +0000 (0:00:00.062) 0:02:42.397 ******* 2021-08-21T14:57:32.082 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-08-21T14:57:32.084 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1629557493.3911078, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1629557493.3871078, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 105181, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1629557493.3871078, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "1956422440", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-08-21T14:57:32.109 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:32 +0000 (0:00:00.225) 0:02:42.622 ******* 2021-08-21T14:57:32.526 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-08-21T14:57:32.548 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:32 +0000 (0:00:00.436) 0:02:43.059 ******* 2021-08-21T14:57:32.586 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi165.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-08-21T14:57:32.612 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:32 +0000 (0:00:00.065) 0:02:43.125 ******* 2021-08-21T14:57:32.638 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-08-21T14:57:32.640 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:32.662 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:32 +0000 (0:00:00.050) 0:02:43.176 ******* 2021-08-21T14:57:32.852 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi165.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-08-21T14:57:32.872 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:32 +0000 (0:00:00.209) 0:02:43.385 ******* 2021-08-21T14:57:33.048 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi165.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-08-21T14:57:33.072 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:33 +0000 (0:00:00.200) 0:02:43.585 ******* 2021-08-21T14:57:33.099 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-08-21T14:57:33.100 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.122 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:33 +0000 (0:00:00.050) 0:02:43.636 ******* 2021-08-21T14:57:33.160 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-08-21T14:57:33.162 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=docker0) => {"ansible_loop_var": "item", "changed": false, "item": "docker0", "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.174 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.175 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.200 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:33 +0000 (0:00:00.078) 0:02:43.714 ******* 2021-08-21T14:57:33.238 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-08-21T14:57:33.240 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'docker0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "docker0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi165.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.250 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.252 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.261 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.263 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.287 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:33 +0000 (0:00:00.086) 0:02:43.800 ******* 2021-08-21T14:57:33.314 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.343 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:33 +0000 (0:00:00.055) 0:02:43.856 ******* 2021-08-21T14:57:33.369 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.394 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:33 +0000 (0:00:00.051) 0:02:43.907 ******* 2021-08-21T14:57:33.421 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-08-21T14:57:33.422 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.446 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:33 +0000 (0:00:00.052) 0:02:43.960 ******* 2021-08-21T14:57:33.474 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-08-21T14:57:33.475 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.500 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:33 +0000 (0:00:00.054) 0:02:44.014 ******* 2021-08-21T14:57:33.527 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-08-21T14:57:33.528 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.552 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:33 +0000 (0:00:00.052) 0:02:44.066 ******* 2021-08-21T14:57:33.579 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-08-21T14:57:33.581 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.606 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:33 +0000 (0:00:00.052) 0:02:44.119 ******* 2021-08-21T14:57:33.632 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-08-21T14:57:33.634 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.660 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:33 +0000 (0:00:00.054) 0:02:44.173 ******* 2021-08-21T14:57:33.689 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:33.717 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:33 +0000 (0:00:00.054) 0:02:44.228 ******* 2021-08-21T14:57:33.935 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-08-21T14:57:33.936 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010467", "end": "2021-08-21 14:57:33.508702", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-08-21 14:57:33.498235", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:57:34.104 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005897", "end": "2021-08-21 14:57:33.679377", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-08-21 14:57:33.673480", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:57:34.262 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005568", "end": "2021-08-21 14:57:33.844673", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-08-21 14:57:33.839105", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-08-21T14:57:34.681 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.255251", "end": "2021-08-21 14:57:34.251249", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-08-21 14:57:33.995998", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:57:34.709 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:34 +0000 (0:00:00.993) 0:02:45.222 ******* 2021-08-21T14:57:34.737 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-08-21T14:57:34.762 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:34 +0000 (0:00:00.053) 0:02:45.275 ******* 2021-08-21T14:57:34.790 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:34.862 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-08-21T14:57:34.893 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:34 +0000 (0:00:00.130) 0:02:45.406 ******* 2021-08-21T14:57:35.904 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-08-21T14:57:35.908 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] 2021-08-21T14:57:35.967 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:35 +0000 (0:00:01.074) 0:02:46.480 ******* 2021-08-21T14:57:36.006 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-08-21T14:57:36.008 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-08-21T14:57:36.030 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:36 +0000 (0:00:00.063) 0:02:46.544 ******* 2021-08-21T14:57:36.058 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-08-21T14:57:36.060 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:36.086 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:36 +0000 (0:00:00.055) 0:02:46.600 ******* 2021-08-21T14:57:36.124 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** ok: [smithi165.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-08-21T14:57:36.148 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:36 +0000 (0:00:00.062) 0:02:46.662 ******* 2021-08-21T14:57:36.928 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** ok: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:57:36.968 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:36 +0000 (0:00:00.812) 0:02:47.474 ******* 2021-08-21T14:57:36.989 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:37.013 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:37 +0000 (0:00:00.052) 0:02:47.527 ******* 2021-08-21T14:57:37.054 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi165.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-08-21T14:57:37.075 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:37 +0000 (0:00:00.062) 0:02:47.589 ******* 2021-08-21T14:57:37.282 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* ok: [smithi165.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-08-21T14:57:37.698 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-08-21T14:57:37.728 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:37 +0000 (0:00:00.652) 0:02:48.242 ******* 2021-08-21T14:57:38.157 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-08-21T14:57:38.159 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "size": 2334, "state": "file", "uid": 0} 2021-08-21T14:57:38.538 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "size": 2334, "state": "file", "uid": 0} 2021-08-21T14:57:38.565 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:38 +0000 (0:00:00.836) 0:02:49.078 ******* 2021-08-21T14:57:39.320 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** ok: [smithi165.front.sepia.ceph.com] => {"cache_update_time": 1629557510, "cache_updated": false, "changed": false} 2021-08-21T14:57:39.342 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:39 +0000 (0:00:00.776) 0:02:49.855 ******* 2021-08-21T14:57:43.571 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": " Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-zj4qy8wu\n", "stderr_lines": [" Running command git clone -q https://github.com/sebastian-philipp/registries-conf-ctl /tmp/pip-req-build-zj4qy8wu"], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-zj4qy8wu\nRequirement already satisfied, skipping upgrade: docopt in /usr/local/lib/python3.8/dist-packages (from registries-conf-ctl==1.0) (0.6.2)\nRequirement already satisfied, skipping upgrade: toml in /usr/local/lib/python3.8/dist-packages (from registries-conf-ctl==1.0) (0.10.2)\nBuilding wheels for collected packages: registries-conf-ctl\n Building wheel for registries-conf-ctl (setup.py): started\n Building wheel for registries-conf-ctl (setup.py): finished with status 'done'\n Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=8069 sha256=28c7228620d6698e5946ba8d366db3782c1afdb5d11ba6060377cb185fb0a19d\n Stored in directory: /tmp/pip-ephem-wheel-cache-otuzeooh/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42\nSuccessfully built registries-conf-ctl\nInstalling collected packages: registries-conf-ctl\n Attempting uninstall: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebasti 2021-08-21T14:57:43.573 INFO:teuthology.task.ansible.out:an-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-req-build-zj4qy8wu", "Requirement already satisfied, skipping upgrade: docopt in /usr/local/lib/python3.8/dist-packages (from registries-conf-ctl==1.0) (0.6.2)", "Requirement already satisfied, skipping upgrade: toml in /usr/local/lib/python3.8/dist-packages (from registries-conf-ctl==1.0) (0.10.2)", "Building wheels for collected packages: registries-conf-ctl", " Building wheel for registries-conf-ctl (setup.py): started", " Building wheel for registries-conf-ctl (setup.py): finished with status 'done'", " Created wheel for registries-conf-ctl: filename=registries_conf_ctl-1.0-py3-none-any.whl size=8069 sha256=28c7228620d6698e5946ba8d366db3782c1afdb5d11ba6060377cb185fb0a19d", " Stored in directory: /tmp/pip-ephem-wheel-cache-otuzeooh/wheels/66/49/be/0ca4a7322f6cd17532451655790e478b4f82776818b44d6d42", "Successfully built registries-conf-ctl", "Installing collected packages: registries-conf-ctl", " Attempting uninstall: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-08-21T14:57:43.594 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:43 +0000 (0:00:04.252) 0:02:54.108 ******* 2021-08-21T14:57:43.781 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* ok: [smithi165.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1629557475.7511084, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "checksum": "76c6651ed619dc361f25958c4bc22bbed7de3326", "ctime": 1628780204.6025991, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 95289, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "application/json", "mode": "0644", "mtime": 1628780204.6025991, "nlink": 1, "path": "/etc/docker/daemon.json", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 73, "uid": 0, "version": "2929411125", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-08-21T14:57:43.808 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:43 +0000 (0:00:00.213) 0:02:54.321 ******* 2021-08-21T14:57:43.856 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-08-21T14:57:43.858 INFO:teuthology.task.ansible.out:skipping: [smithi165.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:57:43.880 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:43 +0000 (0:00:00.072) 0:02:54.394 ******* 2021-08-21T14:57:44.256 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-08-21T14:57:44.258 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.195554", "end": "2021-08-21 14:57:43.836807", "rc": 0, "start": "2021-08-21 14:57:43.641253", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:57:44.281 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:57:44 +0000 (0:00:00.399) 0:02:54.794 ******* 2021-08-21T14:57:45.891 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-08-21T14:57:45.892 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestamp": "Sat 2021-08-21 14:54:44 UTC", "ActiveEnterTimestampMonotonic": "18870352", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket containerd.service sysinit.target docker.socket system.slice network-online.target firewalld.service basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sat 2021-08-21 14:54:38 UTC", "AssertTimestampMonotonic": "13557577", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read", "CleanResult": "success", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sat 2021-08-21 14:54:38 UTC", "ConditionTimestampMonotonic": "13557577", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/ 2021-08-21T14:57:45.894 INFO:teuthology.task.ansible.out:docker.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "yes", "DelegateControllers": "cpu cpuacct cpuset io blkio memory devices pids bpf-firewall bpf-devices", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "858", "ExecMainStartTimestamp": "Sat 2021-08-21 14:54:38 UTC", "ExecMainStartTimestampMonotonic": "13558150", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecReloadEx": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FinalKillSignal": "9", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "IPAccounting": "no", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "Id": "docker.service", "IgnoreOnIso 2021-08-21T14:57:45.896 INFO:teuthology.task.ansible.out:late": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sat 2021-08-21 14:54:38 UTC", "InactiveExitTimestampMonotonic": "13558289", "InvocationID": "1587c2495ac848e3960fc452950bb588", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127777", "LimitSIGPENDINGSoft": "127777", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "858", "MemoryAccounting": "yes", "MemoryCurrent": "138108928", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMPolicy": "continue", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace" 2021-08-21T14:57:45.897 INFO:teuthology.task.ansible.out:, "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectHostname": "no", "ProtectKernelLogs": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "ReloadResult": "success", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target docker.socket", "Restart": "always", "RestartKillSignal": "15", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sat 2021-08-21 14:54:44 UTC", "StateChangeTimestampMonotonic": "18870352", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "16", "TasksMax": "infinity", "TimeoutAbortUSec": "infinity", "TimeoutCleanUSec": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset 2021-08-21T14:57:45.899 INFO:teuthology.task.ansible.out:": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "Wants": "containerd.service network-online.target", "WatchdogSignal": "6", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-08-21T14:57:45.977 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-08-21T14:57:45.978 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-08-21T14:57:45.981 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY [testnodes] *************************************************************** 2021-08-21T14:57:45.997 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Saturday 21 August 2021 14:57:45 +0000 (0:00:01.714) 0:02:56.508 ******* 2021-08-21T14:57:46.994 INFO:teuthology.task.ansible.out:ok: [smithi165.front.sepia.ceph.com] 2021-08-21T14:57:47.014 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** Saturday 21 August 2021 14:57:47 +0000 (0:00:01.018) 0:02:57.527 ******* 2021-08-21T14:57:47.200 INFO:teuthology.task.ansible.out:changed: [smithi165.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-08-21T14:57:47.201 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* 2021-08-21T14:57:47.203 INFO:teuthology.task.ansible.out:smithi165.front.sepia.ceph.com : ok=131 changed=30 unreachable=0 failed=0 skipped=189 rescued=0 ignored=0 Saturday 21 August 2021 14:57:47 +0000 (0:00:00.188) 0:02:57.715 ******* =============================================================================== 2021-08-21T14:57:47.205 INFO:teuthology.task.ansible.out:Create all admin users with sudo access. ------------------------------- 24.24s 2021-08-21T14:57:47.205 INFO:teuthology.task.ansible.out:users : Update authorized_keys using the keys repo --------------------- 21.29s testnode : Install packages -------------------------------------------- 18.42s testnode : Zap all non-root disks --------------------------------------- 8.95s testnode : Ensure packages are not present. ----------------------------- 7.66s Remove revoked users ---------------------------------------------------- 6.73s container-host : Install registries-conf-ctl ---------------------------- 4.25s common : Update apt cache ----------------------------------------------- 3.76s testnode : Install apt keys --------------------------------------------- 2.33s testnode : Install packages via pip ------------------------------------- 2.25s testnode : Update apt cache. -------------------------------------------- 2.22s testnode : Create logical volume(s) ------------------------------------- 2.21s Gathering Facts --------------------------------------------------------- 2.20s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.88s container-host : Restart docker service --------------------------------- 1.71s users : Update authorized_keys for each user with literal keys ---------- 1.58s common : Mask sleep units ----------------------------------------------- 1.26s Gathering Facts --------------------------------------------------------- 1.07s common : Set the hardware clock ----------------------------------------- 1.06s testnode : Install python-pip on apt based systems. --------------------- 1.05s 2021-08-21T14:57:47.615 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi165.front.sepia.ceph.com')] 2021-08-21T14:57:47.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-21T14:57:47.622 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:57:47.677 DEBUG:teuthology.orchestra.run.smithi165:> true 2021-08-21T14:57:47.797 INFO:teuthology.run_tasks:Running task clock... 2021-08-21T14:57:47.808 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-08-21T14:57:47.810 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-08-21T14:57:47.812 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-08-21T14:57:47.848 INFO:teuthology.orchestra.run.smithi165.stdout:21 Aug 14:57:47 ntpd[7950]: ntpd 4.2.8p12@1.3728-o (1): Starting 2021-08-21T14:57:47.849 INFO:teuthology.orchestra.run.smithi165.stdout:21 Aug 14:57:47 ntpd[7950]: Command line: ntpd -gq 2021-08-21T14:57:47.849 INFO:teuthology.orchestra.run.smithi165.stdout:21 Aug 14:57:47 ntpd[7950]: proto: precision = 0.080 usec (-23) 2021-08-21T14:57:47.850 INFO:teuthology.orchestra.run.smithi165.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-08-21T14:57:47.850 INFO:teuthology.orchestra.run.smithi165.stdout:21 Aug 14:57:47 ntpd[7950]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-08-21T14:57:47.851 INFO:teuthology.orchestra.run.smithi165.stderr:restrict ::: KOD does nothing without LIMITED. 2021-08-21T14:57:47.853 INFO:teuthology.orchestra.run.smithi165.stdout:21 Aug 14:57:47 ntpd[7950]: restrict ::: KOD does nothing without LIMITED. 2021-08-21T14:57:47.853 INFO:teuthology.orchestra.run.smithi165.stdout:21 Aug 14:57:47 ntpd[7950]: Listen and drop on 0 v6wildcard [::]:123 2021-08-21T14:57:47.853 INFO:teuthology.orchestra.run.smithi165.stdout:21 Aug 14:57:47 ntpd[7950]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-08-21T14:57:47.854 INFO:teuthology.orchestra.run.smithi165.stdout:21 Aug 14:57:47 ntpd[7950]: Listen normally on 2 lo 127.0.0.1:123 2021-08-21T14:57:47.854 INFO:teuthology.orchestra.run.smithi165.stdout:21 Aug 14:57:47 ntpd[7950]: Listen normally on 3 enp3s0f1 172.21.15.165:123 2021-08-21T14:57:47.855 INFO:teuthology.orchestra.run.smithi165.stdout:21 Aug 14:57:47 ntpd[7950]: Listen normally on 4 lo [::1]:123 2021-08-21T14:57:47.855 INFO:teuthology.orchestra.run.smithi165.stdout:21 Aug 14:57:47 ntpd[7950]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f63%5]:123 2021-08-21T14:57:47.855 INFO:teuthology.orchestra.run.smithi165.stdout:21 Aug 14:57:47 ntpd[7950]: Listening on routing socket on fd #22 for interface updates 2021-08-21T14:57:48.850 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:48 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:48.850 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:48 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:48.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:48 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:48.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:48 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:48.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:48 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:48.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:48 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:48.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:48 ntpd[7950]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-08-21T14:57:48.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:48 ntpd[7950]: can't open /var/log/ntpstats/peerstats.20210821: Permission denied 2021-08-21T14:57:48.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:48 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:48.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:48 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:48.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:48 ntpd[7950]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-08-21T14:57:48.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:48 ntpd[7950]: can't open /var/log/ntpstats/peerstats.20210821: Permission denied 2021-08-21T14:57:49.850 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:49 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:49.850 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:49 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:49.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:49 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:49.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:49 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:49.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:49 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:49.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:49 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:49.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:49 ntpd[7950]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-08-21T14:57:49.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:49 ntpd[7950]: can't open /var/log/ntpstats/peerstats.20210821: Permission denied 2021-08-21T14:57:49.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:49 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:49.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:49 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:49.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:49 ntpd[7950]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-08-21T14:57:49.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:49 ntpd[7950]: can't open /var/log/ntpstats/peerstats.20210821: Permission denied 2021-08-21T14:57:50.850 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:50 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:50.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:50 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:50.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:50 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:50.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:50 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:50.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:50 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:50.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:50 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:50.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:50 ntpd[7950]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-08-21T14:57:50.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:50 ntpd[7950]: can't open /var/log/ntpstats/peerstats.20210821: Permission denied 2021-08-21T14:57:50.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:50 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:50.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:50 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:50.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:50 ntpd[7950]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-08-21T14:57:50.854 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:50 ntpd[7950]: can't open /var/log/ntpstats/peerstats.20210821: Permission denied 2021-08-21T14:57:51.850 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:51 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:51.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:51 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:51.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:51 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:51.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:51 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:51.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:51 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:51.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:51 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:51.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:51 ntpd[7950]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-08-21T14:57:51.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:51 ntpd[7950]: can't open /var/log/ntpstats/peerstats.20210821: Permission denied 2021-08-21T14:57:51.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:51 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:51.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:51 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:51.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:51 ntpd[7950]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-08-21T14:57:51.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:51 ntpd[7950]: can't open /var/log/ntpstats/peerstats.20210821: Permission denied 2021-08-21T14:57:52.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:52 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:52.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:52 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:52.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:52 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:52.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:52 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:52.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:52 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:52.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:52 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:52.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:52 ntpd[7950]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-08-21T14:57:52.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:52 ntpd[7950]: can't open /var/log/ntpstats/peerstats.20210821: Permission denied 2021-08-21T14:57:52.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:52 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:52.854 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:52 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:52.854 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:52 ntpd[7950]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-08-21T14:57:52.854 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:52 ntpd[7950]: can't open /var/log/ntpstats/peerstats.20210821: Permission denied 2021-08-21T14:57:53.850 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:53 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:53.850 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:53 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:53.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:53 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:53.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:53 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:53.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:53 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:53.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:53 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:53.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:53 ntpd[7950]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-08-21T14:57:53.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:53 ntpd[7950]: can't open /var/log/ntpstats/peerstats.20210821: Permission denied 2021-08-21T14:57:53.853 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:53 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:53.854 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:53 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:53.854 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:53 ntpd[7950]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-08-21T14:57:53.854 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:53 ntpd[7950]: can't open /var/log/ntpstats/peerstats.20210821: Permission denied 2021-08-21T14:57:54.850 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:54 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:54.850 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:54 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:54.850 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:54 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:54.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:54 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:54.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:54 ntpd[7950]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-08-21T14:57:54.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:54 ntpd[7950]: can't open /var/log/ntpstats/rawstats.20210821: Permission denied 2021-08-21T14:57:54.851 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:54 ntpd[7950]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-08-21T14:57:54.852 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:54 ntpd[7950]: can't open /var/log/ntpstats/peerstats.20210821: Permission denied 2021-08-21T14:57:54.854 INFO:teuthology.orchestra.run.smithi165.stdout:21 Aug 14:57:54 ntpd[7950]: ntpd: time set +0.393156 s 2021-08-21T14:57:54.854 INFO:teuthology.orchestra.run.smithi165.stdout:ntpd: time set +0.393156s 2021-08-21T14:57:54.854 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:54 ntpd[7950]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-08-21T14:57:54.855 INFO:teuthology.orchestra.run.smithi165.stderr:21 Aug 14:57:54 ntpd[7950]: can't open /var/log/ntpstats/loopstats.20210821: Permission denied 2021-08-21T14:57:54.953 INFO:teuthology.orchestra.run.smithi165.stdout: remote refid st t when poll reach delay offset jitter 2021-08-21T14:57:54.953 INFO:teuthology.orchestra.run.smithi165.stdout:============================================================================== 2021-08-21T14:57:54.954 INFO:teuthology.orchestra.run.smithi165.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-08-21T14:57:54.954 INFO:teuthology.orchestra.run.smithi165.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-08-21T14:57:54.954 INFO:teuthology.orchestra.run.smithi165.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-08-21T14:57:54.954 INFO:teuthology.orchestra.run.smithi165.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-08-21T14:57:54.955 INFO:teuthology.run_tasks:Running task install... 2021-08-21T14:57:54.967 DEBUG:teuthology.task.install:project ceph 2021-08-21T14:57:54.968 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'cae1ffd55a16af4627c36508fde05bf24312fad6'}} 2021-08-21T14:57:54.968 DEBUG:teuthology.task.install:config {'sha1': 'cae1ffd55a16af4627c36508fde05bf24312fad6'} 2021-08-21T14:57:54.968 INFO:teuthology.task.install:Using flavor: basic 2021-08-21T14:57:54.975 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-08-21T14:57:54.976 INFO:teuthology.task.install:extra packages: [] 2021-08-21T14:57:54.977 DEBUG:teuthology.orchestra.run.smithi165:> sudo lsb_release -is 2021-08-21T14:57:55.023 INFO:teuthology.orchestra.run.smithi165.stdout:Ubuntu 2021-08-21T14:57:55.028 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-08-21T14:57:55.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-key list | grep Ceph 2021-08-21T14:57:55.113 INFO:teuthology.orchestra.run.smithi165.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-08-21T14:57:55.168 INFO:teuthology.orchestra.run.smithi165.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-08-21T14:57:55.168 INFO:teuthology.orchestra.run.smithi165.stdout:uid [ unknown] Ceph.com (release key) 2021-08-21T14:57:55.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo lsb_release -is 2021-08-21T14:57:55.215 INFO:teuthology.orchestra.run.smithi165.stdout:Ubuntu 2021-08-21T14:57:55.219 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-08-21T14:57:55.219 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-08-21T14:57:55.220 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=cae1ffd55a16af4627c36508fde05bf24312fad6 2021-08-21T14:57:55.392 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default/ 2021-08-21T14:57:55.393 INFO:teuthology.task.install.deb:Package version is 17.0.0-7135-gcae1ffd5-1focal 2021-08-21T14:57:55.521 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:57:55.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-08-21T14:57:55.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo apt-get update 2021-08-21T14:57:55.734 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2021-08-21T14:57:55.802 INFO:teuthology.orchestra.run.smithi165.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal InRelease 2021-08-21T14:57:55.827 INFO:teuthology.orchestra.run.smithi165.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal Release [7,662 B] 2021-08-21T14:57:55.856 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease 2021-08-21T14:57:55.858 INFO:teuthology.orchestra.run.smithi165.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal Release.gpg 2021-08-21T14:57:55.885 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 Packages [17.1 kB] 2021-08-21T14:57:55.932 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main i386 Packages [3,082 B] 2021-08-21T14:57:55.939 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2021-08-21T14:57:56.023 INFO:teuthology.orchestra.run.smithi165.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2021-08-21T14:57:56.386 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 27.8 kB in 1s (43.6 kB/s) 2021-08-21T14:57:57.165 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2021-08-21T14:57:57.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.0.0-7135-gcae1ffd5-1focal cephadm=17.0.0-7135-gcae1ffd5-1focal ceph-mds=17.0.0-7135-gcae1ffd5-1focal ceph-mgr=17.0.0-7135-gcae1ffd5-1focal ceph-common=17.0.0-7135-gcae1ffd5-1focal ceph-fuse=17.0.0-7135-gcae1ffd5-1focal ceph-test=17.0.0-7135-gcae1ffd5-1focal radosgw=17.0.0-7135-gcae1ffd5-1focal python3-rados=17.0.0-7135-gcae1ffd5-1focal python3-rgw=17.0.0-7135-gcae1ffd5-1focal python3-cephfs=17.0.0-7135-gcae1ffd5-1focal python3-rbd=17.0.0-7135-gcae1ffd5-1focal libcephfs2=17.0.0-7135-gcae1ffd5-1focal libcephfs-dev=17.0.0-7135-gcae1ffd5-1focal librados2=17.0.0-7135-gcae1ffd5-1focal librbd1=17.0.0-7135-gcae1ffd5-1focal rbd-fuse=17.0.0-7135-gcae1ffd5-1focal 2021-08-21T14:57:57.255 INFO:teuthology.orchestra.run.smithi165.stdout:Reading package lists... 2021-08-21T14:57:57.399 INFO:teuthology.orchestra.run.smithi165.stdout:Building dependency tree... 2021-08-21T14:57:57.400 INFO:teuthology.orchestra.run.smithi165.stdout:Reading state information... 2021-08-21T14:57:57.522 INFO:teuthology.orchestra.run.smithi165.stdout:The following packages were automatically installed and are no longer required: 2021-08-21T14:57:57.524 INFO:teuthology.orchestra.run.smithi165.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2021-08-21T14:57:57.524 INFO:teuthology.orchestra.run.smithi165.stdout:Use 'sudo apt autoremove' to remove them. 2021-08-21T14:57:57.524 INFO:teuthology.orchestra.run.smithi165.stdout:The following additional packages will be installed: 2021-08-21T14:57:57.525 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-08-21T14:57:57.525 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq libjq1 2021-08-21T14:57:57.526 INFO:teuthology.orchestra.run.smithi165.stdout: libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d liblttng-ust0 2021-08-21T14:57:57.526 INFO:teuthology.orchestra.run.smithi165.stdout: liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2021-08-21T14:57:57.526 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2 libsqlite3-mod-ceph libxslt1.1 nvme-cli python-pastedeploy-tpl 2021-08-21T14:57:57.526 INFO:teuthology.orchestra.run.smithi165.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2021-08-21T14:57:57.526 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2021-08-21T14:57:57.527 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2021-08-21T14:57:57.527 INFO:teuthology.orchestra.run.smithi165.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2021-08-21T14:57:57.527 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2021-08-21T14:57:57.527 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2021-08-21T14:57:57.528 INFO:teuthology.orchestra.run.smithi165.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2021-08-21T14:57:57.528 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2021-08-21T14:57:57.528 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2021-08-21T14:57:57.528 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2021-08-21T14:57:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-08-21T14:57:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2021-08-21T14:57:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2021-08-21T14:57:57.529 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket python3-webtest python3-werkzeug socat xmlstarlet 2021-08-21T14:57:57.530 INFO:teuthology.orchestra.run.smithi165.stdout:Suggested packages: 2021-08-21T14:57:57.530 INFO:teuthology.orchestra.run.smithi165.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2021-08-21T14:57:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2021-08-21T14:57:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2021-08-21T14:57:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2021-08-21T14:57:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2021-08-21T14:57:57.531 INFO:teuthology.orchestra.run.smithi165.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2021-08-21T14:57:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2021-08-21T14:57:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2021-08-21T14:57:57.532 INFO:teuthology.orchestra.run.smithi165.stdout:Recommended packages: 2021-08-21T14:57:57.532 INFO:teuthology.orchestra.run.smithi165.stdout: btrfs-tools 2021-08-21T14:57:57.618 INFO:teuthology.orchestra.run.smithi165.stdout:The following NEW packages will be installed: 2021-08-21T14:57:57.618 INFO:teuthology.orchestra.run.smithi165.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-08-21T14:57:57.619 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-08-21T14:57:57.619 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2021-08-21T14:57:57.620 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2 libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2021-08-21T14:57:57.620 INFO:teuthology.orchestra.run.smithi165.stdout: liblttng-ust0 liblua5.3-0 liboath0 libonig5 librabbitmq4 libradosstriper1 2021-08-21T14:57:57.620 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libxslt1.1 nvme-cli 2021-08-21T14:57:57.620 INFO:teuthology.orchestra.run.smithi165.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2021-08-21T14:57:57.620 INFO:teuthology.orchestra.run.smithi165.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2021-08-21T14:57:57.621 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2021-08-21T14:57:57.621 INFO:teuthology.orchestra.run.smithi165.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2021-08-21T14:57:57.621 INFO:teuthology.orchestra.run.smithi165.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2021-08-21T14:57:57.621 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2021-08-21T14:57:57.622 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2021-08-21T14:57:57.622 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2021-08-21T14:57:57.622 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2021-08-21T14:57:57.622 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2021-08-21T14:57:57.623 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2021-08-21T14:57:57.623 INFO:teuthology.orchestra.run.smithi165.stdout: python3-simplejson python3-singledispatch python3-sklearn 2021-08-21T14:57:57.623 INFO:teuthology.orchestra.run.smithi165.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2021-08-21T14:57:57.624 INFO:teuthology.orchestra.run.smithi165.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2021-08-21T14:57:57.624 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-08-21T14:57:57.625 INFO:teuthology.orchestra.run.smithi165.stdout:The following packages will be upgraded: 2021-08-21T14:57:57.625 INFO:teuthology.orchestra.run.smithi165.stdout: librados2 librbd1 2021-08-21T14:57:57.768 INFO:teuthology.orchestra.run.smithi165.stdout:2 upgraded, 98 newly installed, 0 to remove and 161 not upgraded. 2021-08-21T14:57:57.768 INFO:teuthology.orchestra.run.smithi165.stdout:Need to get 177 MB of archives. 2021-08-21T14:57:57.769 INFO:teuthology.orchestra.run.smithi165.stdout:After this operation, 830 MB of additional disk space will be used. 2021-08-21T14:57:57.769 INFO:teuthology.orchestra.run.smithi165.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.0.0-7135-gcae1ffd5-1focal [3,471 kB] 2021-08-21T14:57:57.811 INFO:teuthology.orchestra.run.smithi165.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2021-08-21T14:57:58.060 INFO:teuthology.orchestra.run.smithi165.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.0.0-7135-gcae1ffd5-1focal [3,510 kB] 2021-08-21T14:57:58.197 INFO:teuthology.orchestra.run.smithi165.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2021-08-21T14:57:58.204 INFO:teuthology.orchestra.run.smithi165.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2021-08-21T14:57:58.208 INFO:teuthology.orchestra.run.smithi165.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.0.0-7135-gcae1ffd5-1focal [744 kB] 2021-08-21T14:57:58.230 INFO:teuthology.orchestra.run.smithi165.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.0.0-7135-gcae1ffd5-1focal [408 kB] 2021-08-21T14:57:58.242 INFO:teuthology.orchestra.run.smithi165.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2021-08-21T14:57:58.247 INFO:teuthology.orchestra.run.smithi165.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.0.0-7135-gcae1ffd5-1focal [22.4 kB] 2021-08-21T14:57:58.247 INFO:teuthology.orchestra.run.smithi165.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.0.0-7135-gcae1ffd5-1focal [200 kB] 2021-08-21T14:57:58.250 INFO:teuthology.orchestra.run.smithi165.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.0.0-7135-gcae1ffd5-1focal [31.0 kB] 2021-08-21T14:57:58.251 INFO:teuthology.orchestra.run.smithi165.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.0.0-7135-gcae1ffd5-1focal [385 kB] 2021-08-21T14:57:58.256 INFO:teuthology.orchestra.run.smithi165.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2021-08-21T14:57:58.269 INFO:teuthology.orchestra.run.smithi165.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.0.0-7135-gcae1ffd5-1focal [3,917 kB] 2021-08-21T14:57:58.388 INFO:teuthology.orchestra.run.smithi165.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2021-08-21T14:57:58.393 INFO:teuthology.orchestra.run.smithi165.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.0.0-7135-gcae1ffd5-1focal [116 kB] 2021-08-21T14:57:58.397 INFO:teuthology.orchestra.run.smithi165.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.0.0-7135-gcae1ffd5-1focal [477 kB] 2021-08-21T14:57:58.409 INFO:teuthology.orchestra.run.smithi165.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2021-08-21T14:57:58.413 INFO:teuthology.orchestra.run.smithi165.stdout:Get:19 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.0.0-7135-gcae1ffd5-1focal [23.3 MB] 2021-08-21T14:57:58.415 INFO:teuthology.orchestra.run.smithi165.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2021-08-21T14:57:58.443 INFO:teuthology.orchestra.run.smithi165.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2021-08-21T14:57:58.447 INFO:teuthology.orchestra.run.smithi165.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2021-08-21T14:57:58.448 INFO:teuthology.orchestra.run.smithi165.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2021-08-21T14:57:58.462 INFO:teuthology.orchestra.run.smithi165.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jwt all 1.7.1-2ubuntu2 [17.4 kB] 2021-08-21T14:57:58.464 INFO:teuthology.orchestra.run.smithi165.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2021-08-21T14:57:58.466 INFO:teuthology.orchestra.run.smithi165.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2021-08-21T14:57:58.470 INFO:teuthology.orchestra.run.smithi165.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2021-08-21T14:57:58.551 INFO:teuthology.orchestra.run.smithi165.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2021-08-21T14:57:58.552 INFO:teuthology.orchestra.run.smithi165.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2021-08-21T14:57:58.552 INFO:teuthology.orchestra.run.smithi165.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2021-08-21T14:57:58.554 INFO:teuthology.orchestra.run.smithi165.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2021-08-21T14:57:58.555 INFO:teuthology.orchestra.run.smithi165.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2021-08-21T14:57:58.559 INFO:teuthology.orchestra.run.smithi165.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-waitress all 1.4.1-1 [80.6 kB] 2021-08-21T14:57:58.562 INFO:teuthology.orchestra.run.smithi165.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2021-08-21T14:57:58.564 INFO:teuthology.orchestra.run.smithi165.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2021-08-21T14:57:58.581 INFO:teuthology.orchestra.run.smithi165.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2021-08-21T14:57:58.636 INFO:teuthology.orchestra.run.smithi165.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2021-08-21T14:57:58.637 INFO:teuthology.orchestra.run.smithi165.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2021-08-21T14:57:58.638 INFO:teuthology.orchestra.run.smithi165.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2021-08-21T14:57:58.718 INFO:teuthology.orchestra.run.smithi165.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2021-08-21T14:57:58.725 INFO:teuthology.orchestra.run.smithi165.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2021-08-21T14:57:58.730 INFO:teuthology.orchestra.run.smithi165.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2021-08-21T14:57:58.730 INFO:teuthology.orchestra.run.smithi165.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2021-08-21T14:57:58.733 INFO:teuthology.orchestra.run.smithi165.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2021-08-21T14:57:58.737 INFO:teuthology.orchestra.run.smithi165.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2021-08-21T14:57:58.740 INFO:teuthology.orchestra.run.smithi165.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2021-08-21T14:57:58.800 INFO:teuthology.orchestra.run.smithi165.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2021-08-21T14:57:58.801 INFO:teuthology.orchestra.run.smithi165.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2021-08-21T14:57:59.146 INFO:teuthology.orchestra.run.smithi165.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2021-08-21T14:57:59.242 INFO:teuthology.orchestra.run.smithi165.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2021-08-21T14:57:59.311 INFO:teuthology.orchestra.run.smithi165.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2021-08-21T14:57:59.311 INFO:teuthology.orchestra.run.smithi165.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.0.0-7135-gcae1ffd5-1focal [6,402 kB] 2021-08-21T14:57:59.331 INFO:teuthology.orchestra.run.smithi165.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2021-08-21T14:57:59.331 INFO:teuthology.orchestra.run.smithi165.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2021-08-21T14:57:59.332 INFO:teuthology.orchestra.run.smithi165.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2021-08-21T14:57:59.332 INFO:teuthology.orchestra.run.smithi165.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2021-08-21T14:57:59.332 INFO:teuthology.orchestra.run.smithi165.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2021-08-21T14:57:59.333 INFO:teuthology.orchestra.run.smithi165.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2021-08-21T14:57:59.333 INFO:teuthology.orchestra.run.smithi165.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2021-08-21T14:57:59.334 INFO:teuthology.orchestra.run.smithi165.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2021-08-21T14:57:59.406 INFO:teuthology.orchestra.run.smithi165.stdout:Get:61 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.0.0-7135-gcae1ffd5-1focal [196 kB] 2021-08-21T14:57:59.406 INFO:teuthology.orchestra.run.smithi165.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2021-08-21T14:57:59.412 INFO:teuthology.orchestra.run.smithi165.stdout:Get:63 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.0.0-7135-gcae1ffd5-1focal [152 kB] 2021-08-21T14:57:59.416 INFO:teuthology.orchestra.run.smithi165.stdout:Get:64 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.0.0-7135-gcae1ffd5-1focal [1,548 kB] 2021-08-21T14:57:59.445 INFO:teuthology.orchestra.run.smithi165.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2021-08-21T14:57:59.450 INFO:teuthology.orchestra.run.smithi165.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2021-08-21T14:57:59.453 INFO:teuthology.orchestra.run.smithi165.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2021-08-21T14:57:59.456 INFO:teuthology.orchestra.run.smithi165.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2021-08-21T14:57:59.457 INFO:teuthology.orchestra.run.smithi165.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2021-08-21T14:57:59.462 INFO:teuthology.orchestra.run.smithi165.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2021-08-21T14:57:59.463 INFO:teuthology.orchestra.run.smithi165.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2021-08-21T14:57:59.465 INFO:teuthology.orchestra.run.smithi165.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2021-08-21T14:57:59.467 INFO:teuthology.orchestra.run.smithi165.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.0.0-7135-gcae1ffd5-1focal [7,811 kB] 2021-08-21T14:57:59.488 INFO:teuthology.orchestra.run.smithi165.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2021-08-21T14:57:59.550 INFO:teuthology.orchestra.run.smithi165.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2021-08-21T14:57:59.551 INFO:teuthology.orchestra.run.smithi165.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2021-08-21T14:57:59.551 INFO:teuthology.orchestra.run.smithi165.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2021-08-21T14:57:59.554 INFO:teuthology.orchestra.run.smithi165.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.3 [1,382 kB] 2021-08-21T14:57:59.571 INFO:teuthology.orchestra.run.smithi165.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2021-08-21T14:57:59.572 INFO:teuthology.orchestra.run.smithi165.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2021-08-21T14:57:59.573 INFO:teuthology.orchestra.run.smithi165.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2021-08-21T14:57:59.573 INFO:teuthology.orchestra.run.smithi165.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2021-08-21T14:57:59.573 INFO:teuthology.orchestra.run.smithi165.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-py all 1.8.1-1 [65.4 kB] 2021-08-21T14:57:59.653 INFO:teuthology.orchestra.run.smithi165.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2021-08-21T14:57:59.655 INFO:teuthology.orchestra.run.smithi165.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2021-08-21T14:57:59.655 INFO:teuthology.orchestra.run.smithi165.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2021-08-21T14:57:59.657 INFO:teuthology.orchestra.run.smithi165.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2021-08-21T14:57:59.734 INFO:teuthology.orchestra.run.smithi165.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.0.0-7135-gcae1ffd5-1focal [28.5 MB] 2021-08-21T14:58:00.759 INFO:teuthology.orchestra.run.smithi165.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.0.0-7135-gcae1ffd5-1focal [3,812 B] 2021-08-21T14:58:00.773 INFO:teuthology.orchestra.run.smithi165.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.0.0-7135-gcae1ffd5-1focal [850 kB] 2021-08-21T14:58:00.786 INFO:teuthology.orchestra.run.smithi165.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.0.0-7135-gcae1ffd5-1focal [2,349 kB] 2021-08-21T14:58:00.803 INFO:teuthology.orchestra.run.smithi165.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.0.0-7135-gcae1ffd5-1focal [67.7 kB] 2021-08-21T14:58:00.803 INFO:teuthology.orchestra.run.smithi165.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.0.0-7135-gcae1ffd5-1focal [86.1 kB] 2021-08-21T14:58:00.804 INFO:teuthology.orchestra.run.smithi165.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.0.0-7135-gcae1ffd5-1focal [1,592 kB] 2021-08-21T14:58:00.831 INFO:teuthology.orchestra.run.smithi165.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.0.0-7135-gcae1ffd5-1focal [7,876 kB] 2021-08-21T14:58:01.090 INFO:teuthology.orchestra.run.smithi165.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.0.0-7135-gcae1ffd5-1focal [3,928 B] 2021-08-21T14:58:01.091 INFO:teuthology.orchestra.run.smithi165.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.0.0-7135-gcae1ffd5-1focal [48.8 MB] 2021-08-21T14:58:03.026 INFO:teuthology.orchestra.run.smithi165.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.0.0-7135-gcae1ffd5-1focal [20.8 kB] 2021-08-21T14:58:03.026 INFO:teuthology.orchestra.run.smithi165.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.0.0-7135-gcae1ffd5-1focal [11.9 MB] 2021-08-21T14:58:03.298 INFO:teuthology.orchestra.run.smithi165.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.0.0-7135-gcae1ffd5-1focal [78.8 kB] 2021-08-21T14:58:03.887 INFO:teuthology.orchestra.run.smithi165.stdout:Fetched 177 MB in 6s (31.2 MB/s) 2021-08-21T14:58:04.008 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-08-21T14:58:04.061 INFO:teuthology.orchestra.run.smithi165.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138117 files and directories currently installed.) 2021-08-21T14:58:04.066 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../00-liblttng-ust0_2.11.0-1_amd64.deb ... 2021-08-21T14:58:04.144 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2021-08-21T14:58:04.479 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../01-librbd1_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:04.617 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librbd1 (17.0.0-7135-gcae1ffd5-1focal) over (15.2.13-0ubuntu0.20.04.1) ... 2021-08-21T14:58:05.295 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../02-librados2_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:05.437 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librados2 (17.0.0-7135-gcae1ffd5-1focal) over (15.2.13-0ubuntu0.20.04.1) ... 2021-08-21T14:58:06.117 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libcephfs2. 2021-08-21T14:58:06.137 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../03-libcephfs2_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:06.177 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libcephfs2 (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:06.476 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rados. 2021-08-21T14:58:06.492 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../04-python3-rados_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:06.537 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rados (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:06.886 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-08-21T14:58:06.902 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../05-python3-ceph-argparse_17.0.0-7135-gcae1ffd5-1focal_all.deb ... 2021-08-21T14:58:06.947 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-ceph-argparse (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:07.197 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cephfs. 2021-08-21T14:58:07.215 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../06-python3-cephfs_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:07.257 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cephfs (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:07.585 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-ceph-common. 2021-08-21T14:58:07.605 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../07-python3-ceph-common_17.0.0-7135-gcae1ffd5-1focal_all.deb ... 2021-08-21T14:58:07.644 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-ceph-common (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:07.959 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-prettytable. 2021-08-21T14:58:07.975 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../08-python3-prettytable_0.7.2-5_all.deb ... 2021-08-21T14:58:08.034 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2021-08-21T14:58:08.274 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rbd. 2021-08-21T14:58:08.289 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../09-python3-rbd_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:08.332 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rbd (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:08.667 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2021-08-21T14:58:08.685 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../10-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2021-08-21T14:58:08.726 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-08-21T14:58:09.072 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-08-21T14:58:09.089 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../11-librabbitmq4_0.10.0-1_amd64.deb ... 2021-08-21T14:58:09.140 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2021-08-21T14:58:09.460 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-08-21T14:58:09.478 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../12-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2021-08-21T14:58:09.518 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-08-21T14:58:09.860 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package librgw2. 2021-08-21T14:58:09.879 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../13-librgw2_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:09.928 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking librgw2 (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:10.541 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rgw. 2021-08-21T14:58:10.558 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../14-python3-rgw_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:10.601 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rgw (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:10.951 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libleveldb1d:amd64. 2021-08-21T14:58:10.970 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../15-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2021-08-21T14:58:11.010 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-08-21T14:58:11.369 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package liboath0. 2021-08-21T14:58:11.385 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../16-liboath0_2.6.1-1.3_amd64.deb ... 2021-08-21T14:58:11.428 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2021-08-21T14:58:11.744 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libradosstriper1. 2021-08-21T14:58:11.762 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../17-libradosstriper1_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:11.811 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libradosstriper1 (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:12.281 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-common. 2021-08-21T14:58:12.301 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../18-ceph-common_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:12.339 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-common (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:14.699 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-base. 2021-08-21T14:58:14.715 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../19-ceph-base_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:14.768 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-base (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:15.750 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cryptography. 2021-08-21T14:58:15.767 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../20-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2021-08-21T14:58:15.808 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2021-08-21T14:58:16.172 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-openssl. 2021-08-21T14:58:16.187 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../21-python3-openssl_19.0.0-1build1_all.deb ... 2021-08-21T14:58:16.221 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2021-08-21T14:58:16.715 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-08-21T14:58:16.733 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../22-ceph-mgr-modules-core_17.0.0-7135-gcae1ffd5-1focal_all.deb ... 2021-08-21T14:58:16.947 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-modules-core (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:17.370 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-bcrypt. 2021-08-21T14:58:17.387 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../23-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2021-08-21T14:58:17.429 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2021-08-21T14:58:17.747 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cherrypy3. 2021-08-21T14:58:17.763 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../24-python3-cherrypy3_8.9.1-6_all.deb ... 2021-08-21T14:58:17.810 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2021-08-21T14:58:18.145 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-jwt. 2021-08-21T14:58:18.164 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../25-python3-jwt_1.7.1-2ubuntu2_all.deb ... 2021-08-21T14:58:18.204 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2) ... 2021-08-21T14:58:18.553 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-logutils. 2021-08-21T14:58:18.568 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../26-python3-logutils_0.3.3-7_all.deb ... 2021-08-21T14:58:18.613 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-logutils (0.3.3-7) ... 2021-08-21T14:58:18.931 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-markupsafe. 2021-08-21T14:58:18.947 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../27-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2021-08-21T14:58:18.994 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2021-08-21T14:58:19.349 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-mako. 2021-08-21T14:58:19.368 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../28-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2021-08-21T14:58:19.411 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-08-21T14:58:19.745 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-simplegeneric. 2021-08-21T14:58:19.764 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../29-python3-simplegeneric_0.8.1-3_all.deb ... 2021-08-21T14:58:19.804 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2021-08-21T14:58:20.152 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-singledispatch. 2021-08-21T14:58:20.170 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../30-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2021-08-21T14:58:20.206 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2021-08-21T14:58:20.531 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-webob. 2021-08-21T14:58:20.547 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../31-python3-webob_1%3a1.8.5-2_all.deb ... 2021-08-21T14:58:20.591 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2021-08-21T14:58:20.901 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-soupsieve. 2021-08-21T14:58:20.920 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../32-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2021-08-21T14:58:20.960 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2021-08-21T14:58:21.276 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-bs4. 2021-08-21T14:58:21.294 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../33-python3-bs4_4.8.2-1_all.deb ... 2021-08-21T14:58:21.335 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2021-08-21T14:58:21.719 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-waitress. 2021-08-21T14:58:21.735 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../34-python3-waitress_1.4.1-1_all.deb ... 2021-08-21T14:58:21.781 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-waitress (1.4.1-1) ... 2021-08-21T14:58:22.134 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-tempita. 2021-08-21T14:58:22.154 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../35-python3-tempita_0.5.2-6_all.deb ... 2021-08-21T14:58:22.193 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-tempita (0.5.2-6) ... 2021-08-21T14:58:22.505 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-paste. 2021-08-21T14:58:22.520 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../36-python3-paste_3.4.0+dfsg1-1_all.deb ... 2021-08-21T14:58:22.571 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2021-08-21T14:58:22.891 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-08-21T14:58:22.907 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../37-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2021-08-21T14:58:22.951 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2021-08-21T14:58:23.311 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pastedeploy. 2021-08-21T14:58:23.327 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../38-python3-pastedeploy_2.1.0-1_all.deb ... 2021-08-21T14:58:23.377 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2021-08-21T14:58:23.697 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-webtest. 2021-08-21T14:58:23.714 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../39-python3-webtest_2.0.32-2_all.deb ... 2021-08-21T14:58:23.756 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-webtest (2.0.32-2) ... 2021-08-21T14:58:24.067 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pecan. 2021-08-21T14:58:24.084 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../40-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2021-08-21T14:58:24.125 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2021-08-21T14:58:24.480 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-werkzeug. 2021-08-21T14:58:24.498 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../41-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2021-08-21T14:58:24.539 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2021-08-21T14:58:24.917 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2021-08-21T14:58:24.934 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../42-libsqlite3-mod-ceph_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:24.975 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libsqlite3-mod-ceph (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:25.339 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr. 2021-08-21T14:58:25.354 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../43-ceph-mgr_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:25.396 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:25.831 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mon. 2021-08-21T14:58:25.847 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../44-ceph-mon_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:25.881 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mon (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:27.010 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-osd. 2021-08-21T14:58:27.029 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../45-ceph-osd_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:27.068 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-osd (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:30.223 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph. 2021-08-21T14:58:30.242 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../46-ceph_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:30.282 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:30.626 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-fuse. 2021-08-21T14:58:30.643 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../47-ceph-fuse_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:30.685 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-fuse (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:31.138 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mds. 2021-08-21T14:58:31.155 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../48-ceph-mds_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:31.196 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mds (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:31.741 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package cephadm. 2021-08-21T14:58:31.758 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../49-cephadm_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:31.800 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking cephadm (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:32.128 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-jinja2. 2021-08-21T14:58:32.143 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../50-python3-jinja2_2.10.1-2_all.deb ... 2021-08-21T14:58:32.190 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2021-08-21T14:58:32.505 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-08-21T14:58:32.525 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../51-ceph-mgr-cephadm_17.0.0-7135-gcae1ffd5-1focal_all.deb ... 2021-08-21T14:58:32.563 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-cephadm (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:32.909 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-repoze.lru. 2021-08-21T14:58:32.924 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../52-python3-repoze.lru_0.7-2_all.deb ... 2021-08-21T14:58:32.967 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2021-08-21T14:58:33.219 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libjs-underscore. 2021-08-21T14:58:33.227 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../53-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2021-08-21T14:58:33.271 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2021-08-21T14:58:33.539 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libjs-sphinxdoc. 2021-08-21T14:58:33.557 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../54-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2021-08-21T14:58:33.597 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-08-21T14:58:33.950 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-routes. 2021-08-21T14:58:33.971 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../55-python3-routes_2.4.1-2_all.deb ... 2021-08-21T14:58:34.009 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-routes (2.4.1-2) ... 2021-08-21T14:58:34.339 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-08-21T14:58:34.352 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../56-ceph-mgr-dashboard_17.0.0-7135-gcae1ffd5-1focal_all.deb ... 2021-08-21T14:58:34.395 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-dashboard (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:34.877 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-decorator. 2021-08-21T14:58:34.891 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../57-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2021-08-21T14:58:34.931 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2021-08-21T14:58:35.246 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2021-08-21T14:58:35.259 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../58-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2021-08-21T14:58:35.301 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-08-21T14:58:35.647 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-scipy. 2021-08-21T14:58:35.661 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../59-python3-scipy_1.3.3-3build1_amd64.deb ... 2021-08-21T14:58:35.704 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2021-08-21T14:58:37.022 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-08-21T14:58:37.040 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../60-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2021-08-21T14:58:37.080 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-08-21T14:58:37.632 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-joblib. 2021-08-21T14:58:37.642 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../61-python3-joblib_0.14.0-3_all.deb ... 2021-08-21T14:58:37.684 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-joblib (0.14.0-3) ... 2021-08-21T14:58:38.046 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-sklearn. 2021-08-21T14:58:38.062 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../62-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2021-08-21T14:58:38.104 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-08-21T14:58:38.821 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-08-21T14:58:38.841 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../63-ceph-mgr-diskprediction-local_17.0.0-7135-gcae1ffd5-1focal_all.deb ... 2021-08-21T14:58:38.879 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-diskprediction-local (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:40.000 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-cachetools. 2021-08-21T14:58:40.017 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../64-python3-cachetools_4.0.0-1_all.deb ... 2021-08-21T14:58:40.083 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2021-08-21T14:58:40.361 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pyasn1. 2021-08-21T14:58:40.380 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../65-python3-pyasn1_0.4.2-3build1_all.deb ... 2021-08-21T14:58:40.419 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2021-08-21T14:58:40.690 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-08-21T14:58:40.706 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../66-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2021-08-21T14:58:40.774 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-08-21T14:58:41.068 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-rsa. 2021-08-21T14:58:41.084 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../67-python3-rsa_4.0-3_all.deb ... 2021-08-21T14:58:41.127 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-rsa (4.0-3) ... 2021-08-21T14:58:41.430 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-google-auth. 2021-08-21T14:58:41.449 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../68-python3-google-auth_1.5.1-2_all.deb ... 2021-08-21T14:58:41.488 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2021-08-21T14:58:41.841 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-blinker. 2021-08-21T14:58:41.859 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../69-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2021-08-21T14:58:41.907 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-08-21T14:58:42.185 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-oauthlib. 2021-08-21T14:58:42.202 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../70-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2021-08-21T14:58:42.235 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2021-08-21T14:58:42.540 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-requests-oauthlib. 2021-08-21T14:58:42.556 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../71-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2021-08-21T14:58:42.598 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-08-21T14:58:42.859 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-websocket. 2021-08-21T14:58:42.875 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../72-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2021-08-21T14:58:42.910 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2021-08-21T14:58:43.237 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-kubernetes. 2021-08-21T14:58:43.257 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../73-python3-kubernetes_7.0.0~a1-2_all.deb ... 2021-08-21T14:58:43.295 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2021-08-21T14:58:43.867 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-08-21T14:58:43.885 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../74-ceph-mgr-k8sevents_17.0.0-7135-gcae1ffd5-1focal_all.deb ... 2021-08-21T14:58:43.925 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-mgr-k8sevents (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:44.245 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libonig5:amd64. 2021-08-21T14:58:44.261 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../75-libonig5_6.9.4-1_amd64.deb ... 2021-08-21T14:58:44.303 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2021-08-21T14:58:44.689 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libjq1:amd64. 2021-08-21T14:58:44.707 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../76-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-08-21T14:58:44.764 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-08-21T14:58:45.061 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package jq. 2021-08-21T14:58:45.066 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../77-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2021-08-21T14:58:45.108 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2021-08-21T14:58:45.370 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package socat. 2021-08-21T14:58:45.386 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../78-socat_1.7.3.3-2_amd64.deb ... 2021-08-21T14:58:45.428 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking socat (1.7.3.3-2) ... 2021-08-21T14:58:45.806 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-08-21T14:58:45.825 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../79-libxslt1.1_1.1.34-4_amd64.deb ... 2021-08-21T14:58:45.865 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2021-08-21T14:58:46.143 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package xmlstarlet. 2021-08-21T14:58:46.161 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../80-xmlstarlet_1.6.1-2build1_amd64.deb ... 2021-08-21T14:58:46.201 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2021-08-21T14:58:46.649 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package ceph-test. 2021-08-21T14:58:46.664 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../81-ceph-test_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:46.707 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking ceph-test (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:51.186 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package libcephfs-dev. 2021-08-21T14:58:51.208 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../82-libcephfs-dev_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:51.244 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking libcephfs-dev (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:51.623 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package nvme-cli. 2021-08-21T14:58:51.639 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../83-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2021-08-21T14:58:51.699 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2021-08-21T14:58:52.070 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-atomicwrites. 2021-08-21T14:58:52.089 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../84-python3-atomicwrites_1.1.5-2build1_all.deb ... 2021-08-21T14:58:52.123 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2021-08-21T14:58:52.431 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-attr. 2021-08-21T14:58:52.447 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../85-python3-attr_19.3.0-2_all.deb ... 2021-08-21T14:58:52.514 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-attr (19.3.0-2) ... 2021-08-21T14:58:52.843 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-webencodings. 2021-08-21T14:58:52.859 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../86-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2021-08-21T14:58:52.901 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2021-08-21T14:58:53.154 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-html5lib. 2021-08-21T14:58:53.173 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../87-python3-html5lib_1.0.1-2_all.deb ... 2021-08-21T14:58:53.203 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2021-08-21T14:58:53.490 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-08-21T14:58:53.509 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../88-python3-lxml_4.5.0-1ubuntu0.3_amd64.deb ... 2021-08-21T14:58:53.540 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.3) ... 2021-08-21T14:58:54.045 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-packaging. 2021-08-21T14:58:54.065 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../89-python3-packaging_20.3-1_all.deb ... 2021-08-21T14:58:54.103 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-packaging (20.3-1) ... 2021-08-21T14:58:54.390 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pastescript. 2021-08-21T14:58:54.409 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../90-python3-pastescript_2.0.2-4_all.deb ... 2021-08-21T14:58:54.448 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2021-08-21T14:58:54.792 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pluggy. 2021-08-21T14:58:54.812 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../91-python3-pluggy_0.13.0-2_all.deb ... 2021-08-21T14:58:54.850 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2021-08-21T14:58:55.145 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-psutil. 2021-08-21T14:58:55.165 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../92-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2021-08-21T14:58:55.203 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2021-08-21T14:58:55.540 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-py. 2021-08-21T14:58:55.556 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../93-python3-py_1.8.1-1_all.deb ... 2021-08-21T14:58:55.623 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-py (1.8.1-1) ... 2021-08-21T14:58:55.942 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pyinotify. 2021-08-21T14:58:55.959 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../94-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2021-08-21T14:58:56.001 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-08-21T14:58:56.320 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-wcwidth. 2021-08-21T14:58:56.336 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../95-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2021-08-21T14:58:56.383 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2021-08-21T14:58:56.681 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-pytest. 2021-08-21T14:58:56.699 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../96-python3-pytest_4.6.9-1_all.deb ... 2021-08-21T14:58:56.739 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-pytest (4.6.9-1) ... 2021-08-21T14:58:57.093 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package python3-simplejson. 2021-08-21T14:58:57.114 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../97-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2021-08-21T14:58:57.175 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2021-08-21T14:58:57.496 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package radosgw. 2021-08-21T14:58:57.514 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../98-radosgw_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:57.554 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking radosgw (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:58.815 INFO:teuthology.orchestra.run.smithi165.stdout:Selecting previously unselected package rbd-fuse. 2021-08-21T14:58:58.833 INFO:teuthology.orchestra.run.smithi165.stdout:Preparing to unpack .../99-rbd-fuse_17.0.0-7135-gcae1ffd5-1focal_amd64.deb ... 2021-08-21T14:58:58.873 INFO:teuthology.orchestra.run.smithi165.stdout:Unpacking rbd-fuse (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:58:59.175 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-attr (19.3.0-2) ... 2021-08-21T14:58:59.520 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2021-08-21T14:58:59.645 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2021-08-21T14:58:59.789 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2021-08-21T14:58:59.989 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up cephadm (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:00.617 INFO:teuthology.orchestra.run.smithi165.stdout:Adding system user cephadm....done 2021-08-21T14:59:00.795 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-waitress (1.4.1-1) ... 2021-08-21T14:59:01.130 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-repoze.lru (0.7-2) ... 2021-08-21T14:59:01.423 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up liboath0 (2.6.1-1.3) ... 2021-08-21T14:59:01.550 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-py (1.8.1-1) ... 2021-08-21T14:59:01.883 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-joblib (0.14.0-3) ... 2021-08-21T14:59:02.252 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-jwt (1.7.1-2ubuntu2) ... 2021-08-21T14:59:02.545 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cachetools (4.0.0-1) ... 2021-08-21T14:59:02.837 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2021-08-21T14:59:03.117 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2021-08-21T14:59:03.406 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-ceph-argparse (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:03.707 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2021-08-21T14:59:03.850 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2021-08-21T14:59:04.126 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2021-08-21T14:59:04.427 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2021-08-21T14:59:04.795 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2021-08-21T14:59:05.096 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2021-08-21T14:59:05.405 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2021-08-21T14:59:05.707 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2021-08-21T14:59:06.032 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-tempita (0.5.2-6) ... 2021-08-21T14:59:06.317 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-packaging (20.3-1) ... 2021-08-21T14:59:06.643 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2021-08-21T14:59:06.769 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2021-08-21T14:59:07.062 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2021-08-21T14:59:07.455 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2021-08-21T14:59:07.831 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2021-08-21T14:59:08.208 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up socat (1.7.3.3-2) ... 2021-08-21T14:59:08.346 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-ceph-common (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:08.644 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-html5lib (1.0.1-2) ... 2021-08-21T14:59:08.987 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2021-08-21T14:59:09.113 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pluggy (0.13.0-2) ... 2021-08-21T14:59:09.462 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2021-08-21T14:59:09.590 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2021-08-21T14:59:09.724 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2021-08-21T14:59:10.051 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2021-08-21T14:59:10.327 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-logutils (0.3.3-7) ... 2021-08-21T14:59:10.628 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2021-08-21T14:59:10.754 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2021-08-21T14:59:11.039 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-prettytable (0.7.2-5) ... 2021-08-21T14:59:11.340 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2021-08-21T14:59:11.649 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2021-08-21T14:59:12.063 INFO:teuthology.orchestra.run.smithi165.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-08-21T14:59:12.302 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2021-08-21T14:59:12.428 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2021-08-21T14:59:12.762 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2021-08-21T14:59:13.122 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2021-08-21T14:59:13.248 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2021-08-21T14:59:13.374 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-webob (1:1.8.5-2) ... 2021-08-21T14:59:13.725 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2021-08-21T14:59:14.018 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librados2 (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:14.194 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2021-08-21T14:59:15.657 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librgw2 (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:15.792 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libsqlite3-mod-ceph (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:15.918 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2021-08-21T14:59:16.044 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pytest (4.6.9-1) ... 2021-08-21T14:59:16.421 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2021-08-21T14:59:16.730 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2021-08-21T14:59:17.023 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-bs4 (4.8.2-1) ... 2021-08-21T14:59:17.340 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libcephfs2 (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:17.483 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2021-08-21T14:59:17.793 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libradosstriper1 (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:17.919 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2021-08-21T14:59:18.237 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2021-08-21T14:59:19.283 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up librbd1 (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:19.500 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2021-08-21T14:59:19.726 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2021-08-21T14:59:19.852 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.3) ... 2021-08-21T14:59:20.162 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2021-08-21T14:59:20.304 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rsa (4.0-3) ... 2021-08-21T14:59:20.581 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-webtest (2.0.32-2) ... 2021-08-21T14:59:20.873 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-modules-core (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:21.016 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-fuse (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:21.242 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-08-21T14:59:21.242 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-08-21T14:59:21.684 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up libcephfs-dev (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:21.835 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2021-08-21T14:59:22.136 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pastescript (2.0.2-4) ... 2021-08-21T14:59:22.446 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2021-08-21T14:59:22.790 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rados (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:22.890 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rbd (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:22.991 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up rbd-fuse (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:23.117 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-routes (2.4.1-2) ... 2021-08-21T14:59:23.411 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-google-auth (1.5.1-2) ... 2021-08-21T14:59:23.806 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-rgw (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:23.932 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-cephfs (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:24.075 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2021-08-21T14:59:25.219 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-common (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:25.780 INFO:teuthology.orchestra.run.smithi165.stdout:Adding group ceph....done 2021-08-21T14:59:26.157 INFO:teuthology.orchestra.run.smithi165.stdout:Adding system user ceph....done 2021-08-21T14:59:26.332 INFO:teuthology.orchestra.run.smithi165.stdout:Setting system user ceph properties....done 2021-08-21T14:59:26.377 INFO:teuthology.orchestra.run.smithi165.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2021-08-21T14:59:26.528 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-08-21T14:59:26.814 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-08-21T14:59:27.279 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-test (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:27.405 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up radosgw (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:27.678 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-08-21T14:59:27.679 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-08-21T14:59:28.339 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-base (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:28.689 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-08-21T14:59:29.124 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mds (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:29.359 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-08-21T14:59:29.360 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-08-21T14:59:29.776 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:30.007 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-08-21T14:59:30.007 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-08-21T14:59:30.485 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-osd (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:31.046 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-08-21T14:59:31.047 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-08-21T14:59:31.487 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-k8sevents (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:31.589 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-diskprediction-local (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:31.714 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mon (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:31.937 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-08-21T14:59:31.938 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-08-21T14:59:32.398 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-cephadm (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:32.567 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:32.692 INFO:teuthology.orchestra.run.smithi165.stdout:Setting up ceph-mgr-dashboard (17.0.0-7135-gcae1ffd5-1focal) ... 2021-08-21T14:59:32.836 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for systemd (245.4-4ubuntu3.4) ... 2021-08-21T14:59:33.128 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for man-db (2.9.1-1) ... 2021-08-21T14:59:36.164 INFO:teuthology.orchestra.run.smithi165.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.2) ... 2021-08-21T14:59:36.571 INFO:teuthology.orchestra.run.smithi165.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-08-21T14:59:36.574 DEBUG:teuthology.parallel:result is None 2021-08-21T14:59:36.575 DEBUG:teuthology.orchestra.run.smithi165:> sudo lsb_release -is 2021-08-21T14:59:36.621 INFO:teuthology.orchestra.run.smithi165.stdout:Ubuntu 2021-08-21T14:59:36.626 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-08-21T14:59:36.627 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=cae1ffd55a16af4627c36508fde05bf24312fad6 2021-08-21T14:59:36.797 DEBUG:teuthology.orchestra.run.smithi165:> dpkg-query -W -f '${Version}' ceph 2021-08-21T14:59:36.818 INFO:teuthology.orchestra.run.smithi165.stdout:17.0.0-7135-gcae1ffd5-1focal 2021-08-21T14:59:36.819 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-7135-gcae1ffd5-1focal 2021-08-21T14:59:36.820 INFO:teuthology.task.install:The correct ceph version 17.0.0-7135-gcae1ffd5-1focal is installed. 2021-08-21T14:59:36.820 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-08-21T14:59:36.821 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:59:36.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-08-21T14:59:36.831 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-08-21T14:59:36.832 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:59:36.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/daemon-helper 2021-08-21T14:59:36.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-08-21T14:59:36.852 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-08-21T14:59:36.853 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:59:36.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/adjust-ulimits 2021-08-21T14:59:36.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-08-21T14:59:36.919 INFO:teuthology.run_tasks:Running task ceph... 2021-08-21T14:59:36.980 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-08-21T14:59:36.981 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /var/log/ceph 2021-08-21T14:59:36.994 INFO:tasks.ceph:Disabling ceph logrotate... 2021-08-21T14:59:36.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/logrotate.d/ceph 2021-08-21T14:59:37.053 INFO:tasks.ceph:Creating extra log directories... 2021-08-21T14:59:37.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-08-21T14:59:37.072 INFO:tasks.ceph:Creating ceph cluster ceph... 2021-08-21T14:59:37.073 INFO:tasks.ceph:config {'conf': {'osd': {'osd min pg log entries': 5, 'osd_fast_fail_on_connection_refused': False, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-08-21T14:59:37.073 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2021-08-21_14:21:15-rados-wip-yuri-master-8.20.21-distro-basic-smithi/6350476', 'branch': 'wip-yuri-master-8.20.21', 'description': 'rados/singleton/{all/osd-recovery-incomplete mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '6350476', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2021-08-21_14:21:15-rados-wip-yuri-master-8.20.21-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri-master-8.20.21'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'cae1ffd55a16af4627c36508fde05bf24312fad6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'cae1ffd55a16af4627c36508fde05bf24312fad6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri-master-8.20.21', 'sha1': 'cae1ffd55a16af4627c36508fde05bf24312fad6'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 51, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'sha1': 'cae1ffd55a16af4627c36508fde05bf24312fad6', 'sleep_before_teardown': 0, 'suite': 'rados', 'suite_branch': 'wip-yuri-master-8.20.21', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_cae1ffd55a16af4627c36508fde05bf24312fad6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'cae1ffd55a16af4627c36508fde05bf24312fad6', 'targets': {'smithi165.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNjqTYpGqmRw5Cy5aurn0f7qY/DYXAUcAWc1XqSqr2zsn3Omu0698Q7pGEs/T4d2ZfSm5koCzjJtYm+DQGdOhPM2cpNichQvhJaE0BCaW5Y5TJcbNDdWp+WOH/9yZOe4NJnS7Jh+I75j6hJemGJIZRd71a7mumxgaKbhtfmseVpcdOzX3YhVBwnzwS/SQaeO8YIlON7lMwv1EwNI7l0F9PVx4F+f9OBZsHamWlKTvs/daugwOmeOSSpcoQZQPZ9Zw8Wmh5T+YfZtqyW/EDhuBg01t2o1BSq0gRRuWpBenT7ax2qKw99xHU+QXpiCk6oM7lbqvZ9ANLgOq0MB5yLxA7PG8EhTnq81QzQcpudqeUl25wdIWe7BWb5lHdKG4dlqm8BoKPMSOW8fUd7C6ZDNab9vORjwqPW2ra9aX7894l5S8a/GiAx1gwBStBF6LntI+yk01kmMMUjljwV0K0DK9yOYbEoWstOmgniJNSk8JmqxpWFSwkmROuMIpqlfDKcp0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd min pg log entries': 5, 'osd_fast_fail_on_connection_refused': False, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'fs': 'xfs', 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'cae1ffd55a16af4627c36508fde05bf24312fad6', 'cluster': 'ceph'}}, {'osd_recovery.test_incomplete_pgs': None}], 'teuthology_branch': 'master', 'teuthology_sha1': '321319b12ea4ff9b63c7655015a3156de2c3b279', 'timestamp': '2021-08-21_14:21:15', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-08-21T14:59:37.074 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-08-21T14:59:37.080 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m0777 -- /var/run/ceph 2021-08-21T14:59:37.138 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:59:37.139 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2021-08-21T14:59:37.144 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-08-21T14:59:37.144 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-08-21T14:59:37.153 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-08-21T14:59:37.153 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-08-21T14:59:37.153 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 6h/6d Inode: 348 Links: 1 2021-08-21T14:59:37.154 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-08-21T14:59:37.154 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2021-08-21 14:57:19.055996236 +0000 2021-08-21T14:59:37.154 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2021-08-21 14:57:18.899996240 +0000 2021-08-21T14:59:37.154 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2021-08-21 14:57:18.899996240 +0000 2021-08-21T14:59:37.155 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2021-08-21T14:59:37.163 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2021-08-21T14:59:37.164 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2021-08-21T14:59:37.164 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000417547 s, 1.2 MB/s 2021-08-21T14:59:37.168 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-08-21T14:59:37.176 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-08-21T14:59:37.176 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-08-21T14:59:37.176 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 6h/6d Inode: 353 Links: 1 2021-08-21T14:59:37.177 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-08-21T14:59:37.177 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2021-08-21 14:57:19.451996228 +0000 2021-08-21T14:59:37.177 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2021-08-21 14:57:19.287996231 +0000 2021-08-21T14:59:37.178 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2021-08-21 14:57:19.287996231 +0000 2021-08-21T14:59:37.178 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2021-08-21T14:59:37.183 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2021-08-21T14:59:37.184 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2021-08-21T14:59:37.184 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000400531 s, 1.3 MB/s 2021-08-21T14:59:37.189 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-08-21T14:59:37.196 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-08-21T14:59:37.197 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-08-21T14:59:37.197 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 6h/6d Inode: 360 Links: 1 2021-08-21T14:59:37.197 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-08-21T14:59:37.198 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2021-08-21 14:57:19.819996219 +0000 2021-08-21T14:59:37.198 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2021-08-21 14:57:19.643996223 +0000 2021-08-21T14:59:37.198 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2021-08-21 14:57:19.643996223 +0000 2021-08-21T14:59:37.198 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2021-08-21T14:59:37.204 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2021-08-21T14:59:37.205 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2021-08-21T14:59:37.206 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000388917 s, 1.3 MB/s 2021-08-21T14:59:37.209 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-08-21T14:59:37.216 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-08-21T14:59:37.217 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-08-21T14:59:37.217 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 6h/6d Inode: 364 Links: 1 2021-08-21T14:59:37.217 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-08-21T14:59:37.217 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2021-08-21 14:57:20.199996211 +0000 2021-08-21T14:59:37.217 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2021-08-21 14:57:20.039996214 +0000 2021-08-21T14:59:37.218 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2021-08-21 14:57:20.039996214 +0000 2021-08-21T14:59:37.218 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: - 2021-08-21T14:59:37.223 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2021-08-21T14:59:37.224 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2021-08-21T14:59:37.224 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.00040803 s, 1.3 MB/s 2021-08-21T14:59:37.228 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-08-21T14:59:37.228 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi165.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2021-08-21T14:59:37.229 INFO:tasks.ceph:Generating config... 2021-08-21T14:59:37.230 INFO:tasks.ceph:[osd] osd min pg log entries = 5 2021-08-21T14:59:37.231 INFO:tasks.ceph:[osd] osd_fast_fail_on_connection_refused = False 2021-08-21T14:59:37.231 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2021-08-21T14:59:37.231 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2021-08-21T14:59:37.231 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2021-08-21T14:59:37.232 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2021-08-21T14:59:37.232 INFO:tasks.ceph:[osd] debug bluefs = 20 2021-08-21T14:59:37.232 INFO:tasks.ceph:[osd] debug bluestore = 20 2021-08-21T14:59:37.233 INFO:tasks.ceph:[osd] debug ms = 1 2021-08-21T14:59:37.233 INFO:tasks.ceph:[osd] debug osd = 20 2021-08-21T14:59:37.233 INFO:tasks.ceph:[osd] debug rocksdb = 10 2021-08-21T14:59:37.234 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2021-08-21T14:59:37.234 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2021-08-21T14:59:37.234 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2021-08-21T14:59:37.235 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2021-08-21T14:59:37.235 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2021-08-21T14:59:37.236 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2021-08-21T14:59:37.236 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2021-08-21T14:59:37.236 INFO:tasks.ceph:[osd] osd op queue = debug_random 2021-08-21T14:59:37.237 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2021-08-21T14:59:37.237 INFO:tasks.ceph:[global] mon election default strategy = 3 2021-08-21T14:59:37.237 INFO:tasks.ceph:[global] ms bind msgr1 = False 2021-08-21T14:59:37.238 INFO:tasks.ceph:[global] ms bind msgr2 = True 2021-08-21T14:59:37.238 INFO:tasks.ceph:[global] ms type = async 2021-08-21T14:59:37.238 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-08-21T14:59:37.238 INFO:tasks.ceph:[mgr] debug ms = 1 2021-08-21T14:59:37.239 INFO:tasks.ceph:[mon] debug mon = 20 2021-08-21T14:59:37.239 INFO:tasks.ceph:[mon] debug ms = 1 2021-08-21T14:59:37.239 INFO:tasks.ceph:[mon] debug paxos = 20 2021-08-21T14:59:37.240 INFO:tasks.ceph:[mon] mon scrub interval = 300 2021-08-21T14:59:37.240 INFO:tasks.ceph:Setting up mon.a... 2021-08-21T14:59:37.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2021-08-21T14:59:37.313 INFO:teuthology.orchestra.run.smithi165.stdout:creating /etc/ceph/ceph.keyring 2021-08-21T14:59:37.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2021-08-21T14:59:37.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-08-21T14:59:37.364 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.165'), ('mon.b', '[v2:172.21.15.165:3301,v1:172.21.15.165:6790]'), ('mon.c', '[v2:172.21.15.165:3302,v1:172.21.15.165:6791]')] 2021-08-21T14:59:37.365 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.165,[v2:172.21.15.165:3301,v1:172.21.15.165:6790],[v2:172.21.15.165:3302,v1:172.21.15.165:6791]', 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd min pg log entries': 5, 'osd_fast_fail_on_connection_refused': False, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2021-08-21T14:59:37.366 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:59:37.366 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-08-21T14:59:37.371 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.165 --addv b '[v2:172.21.15.165:3301,v1:172.21.15.165:6790]' --addv c '[v2:172.21.15.165:3302,v1:172.21.15.165:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2021-08-21T14:59:37.452 INFO:teuthology.orchestra.run.smithi165.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2021-08-21T14:59:37.452 INFO:teuthology.orchestra.run.smithi165.stdout:monmaptool: generated fsid b82e8492-200f-4d81-953f-d1ffd38e4baa 2021-08-21T14:59:37.453 INFO:teuthology.orchestra.run.smithi165.stdout:setting min_mon_release = octopus 2021-08-21T14:59:37.453 INFO:teuthology.orchestra.run.smithi165.stdout:epoch 0 2021-08-21T14:59:37.453 INFO:teuthology.orchestra.run.smithi165.stdout:fsid b82e8492-200f-4d81-953f-d1ffd38e4baa 2021-08-21T14:59:37.453 INFO:teuthology.orchestra.run.smithi165.stdout:last_changed 2021-08-21T14:59:37.453856+0000 2021-08-21T14:59:37.453 INFO:teuthology.orchestra.run.smithi165.stdout:created 2021-08-21T14:59:37.453856+0000 2021-08-21T14:59:37.454 INFO:teuthology.orchestra.run.smithi165.stdout:min_mon_release 15 (octopus) 2021-08-21T14:59:37.454 INFO:teuthology.orchestra.run.smithi165.stdout:election_strategy: 3 2021-08-21T14:59:37.454 INFO:teuthology.orchestra.run.smithi165.stdout:0: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.a 2021-08-21T14:59:37.454 INFO:teuthology.orchestra.run.smithi165.stdout:1: [v2:172.21.15.165:3301/0,v1:172.21.15.165:6790/0] mon.b 2021-08-21T14:59:37.455 INFO:teuthology.orchestra.run.smithi165.stdout:2: [v2:172.21.15.165:3302/0,v1:172.21.15.165:6791/0] mon.c 2021-08-21T14:59:37.455 INFO:teuthology.orchestra.run.smithi165.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2021-08-21T14:59:37.456 DEBUG:teuthology.orchestra.run.smithi165:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-08-21T14:59:37.461 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b82e8492-200f-4d81-953f-d1ffd38e4baa... 2021-08-21T14:59:37.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-08-21T14:59:37.537 INFO:teuthology.orchestra.run.smithi165.stdout:[global] 2021-08-21T14:59:37.538 INFO:teuthology.orchestra.run.smithi165.stdout: chdir = "" 2021-08-21T14:59:37.538 INFO:teuthology.orchestra.run.smithi165.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-08-21T14:59:37.538 INFO:teuthology.orchestra.run.smithi165.stdout: auth supported = cephx 2021-08-21T14:59:37.539 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.539 INFO:teuthology.orchestra.run.smithi165.stdout: filestore xattr use omap = true 2021-08-21T14:59:37.539 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.539 INFO:teuthology.orchestra.run.smithi165.stdout: mon clock drift allowed = 1.000 2021-08-21T14:59:37.539 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.540 INFO:teuthology.orchestra.run.smithi165.stdout: osd crush chooseleaf type = 0 2021-08-21T14:59:37.540 INFO:teuthology.orchestra.run.smithi165.stdout: auth debug = true 2021-08-21T14:59:37.540 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.540 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on old message = true 2021-08-21T14:59:37.540 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on bug = true 2021-08-21T14:59:37.541 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.541 INFO:teuthology.orchestra.run.smithi165.stdout: mon max pg per osd = 10000 # >= luminous 2021-08-21T14:59:37.541 INFO:teuthology.orchestra.run.smithi165.stdout: mon pg warn max object skew = 0 2021-08-21T14:59:37.541 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.542 INFO:teuthology.orchestra.run.smithi165.stdout: # disable pg_autoscaler by default for new pools 2021-08-21T14:59:37.542 INFO:teuthology.orchestra.run.smithi165.stdout: osd_pool_default_pg_autoscale_mode = off 2021-08-21T14:59:37.542 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.542 INFO:teuthology.orchestra.run.smithi165.stdout: osd pool default size = 2 2021-08-21T14:59:37.542 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.543 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd allow primary affinity = true 2021-08-21T14:59:37.543 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd allow pg remap = true 2021-08-21T14:59:37.543 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on legacy crush tunables = false 2021-08-21T14:59:37.543 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on crush straw calc version zero = false 2021-08-21T14:59:37.544 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on no sortbitwise = false 2021-08-21T14:59:37.544 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on osd down out interval zero = false 2021-08-21T14:59:37.544 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on too few osds = false 2021-08-21T14:59:37.544 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-08-21T14:59:37.544 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_pool_no_redundancy = false 2021-08-21T14:59:37.545 INFO:teuthology.orchestra.run.smithi165.stdout: mon_allow_pool_size_one = true 2021-08-21T14:59:37.545 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.545 INFO:teuthology.orchestra.run.smithi165.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-08-21T14:59:37.545 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.546 INFO:teuthology.orchestra.run.smithi165.stdout: osd default data pool replay window = 5 2021-08-21T14:59:37.546 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.546 INFO:teuthology.orchestra.run.smithi165.stdout: mon allow pool delete = true 2021-08-21T14:59:37.546 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.546 INFO:teuthology.orchestra.run.smithi165.stdout: mon cluster log file level = debug 2021-08-21T14:59:37.547 INFO:teuthology.orchestra.run.smithi165.stdout: debug asserts on shutdown = true 2021-08-21T14:59:37.547 INFO:teuthology.orchestra.run.smithi165.stdout: mon health detail to clog = false 2021-08-21T14:59:37.547 INFO:teuthology.orchestra.run.smithi165.stdout: mon host = "172.21.15.165,[v2:172.21.15.165:3301,v1:172.21.15.165:6790],[v2:172.21.15.165:3302,v1:172.21.15.165:6791]" 2021-08-21T14:59:37.547 INFO:teuthology.orchestra.run.smithi165.stdout: mon election default strategy = 3 2021-08-21T14:59:37.548 INFO:teuthology.orchestra.run.smithi165.stdout: ms bind msgr1 = False 2021-08-21T14:59:37.548 INFO:teuthology.orchestra.run.smithi165.stdout: ms bind msgr2 = True 2021-08-21T14:59:37.548 INFO:teuthology.orchestra.run.smithi165.stdout: ms type = async 2021-08-21T14:59:37.548 INFO:teuthology.orchestra.run.smithi165.stdout: fsid = b82e8492-200f-4d81-953f-d1ffd38e4baa 2021-08-21T14:59:37.548 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.549 INFO:teuthology.orchestra.run.smithi165.stdout:[osd] 2021-08-21T14:59:37.549 INFO:teuthology.orchestra.run.smithi165.stdout: osd journal size = 100 2021-08-21T14:59:37.549 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.549 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub load threshold = 5.0 2021-08-21T14:59:37.550 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub max interval = 600 2021-08-21T14:59:37.550 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.550 INFO:teuthology.orchestra.run.smithi165.stdout: osd recover clone overlap = true 2021-08-21T14:59:37.550 INFO:teuthology.orchestra.run.smithi165.stdout: osd recovery max chunk = 1048576 2021-08-21T14:59:37.551 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.551 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug shutdown = true 2021-08-21T14:59:37.551 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug op order = true 2021-08-21T14:59:37.551 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug verify stray on activate = true 2021-08-21T14:59:37.551 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.552 INFO:teuthology.orchestra.run.smithi165.stdout: osd open classes on start = true 2021-08-21T14:59:37.552 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug pg log writeout = true 2021-08-21T14:59:37.552 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.552 INFO:teuthology.orchestra.run.smithi165.stdout: osd deep scrub update digest min age = 30 2021-08-21T14:59:37.552 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.553 INFO:teuthology.orchestra.run.smithi165.stdout: osd map max advance = 10 2021-08-21T14:59:37.553 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.553 INFO:teuthology.orchestra.run.smithi165.stdout: journal zero on create = true 2021-08-21T14:59:37.553 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.554 INFO:teuthology.orchestra.run.smithi165.stdout: filestore ondisk finisher threads = 3 2021-08-21T14:59:37.554 INFO:teuthology.orchestra.run.smithi165.stdout: filestore apply finisher threads = 3 2021-08-21T14:59:37.554 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.554 INFO:teuthology.orchestra.run.smithi165.stdout: bdev debug aio = true 2021-08-21T14:59:37.554 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug misdirected ops = true 2021-08-21T14:59:37.555 INFO:teuthology.orchestra.run.smithi165.stdout: osd min pg log entries = 5 2021-08-21T14:59:37.555 INFO:teuthology.orchestra.run.smithi165.stdout: osd_fast_fail_on_connection_refused = False 2021-08-21T14:59:37.555 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore block size = 96636764160 2021-08-21T14:59:37.555 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore compression algorithm = snappy 2021-08-21T14:59:37.556 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore compression mode = aggressive 2021-08-21T14:59:37.556 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore fsck on mount = True 2021-08-21T14:59:37.556 INFO:teuthology.orchestra.run.smithi165.stdout: debug bluefs = 20 2021-08-21T14:59:37.556 INFO:teuthology.orchestra.run.smithi165.stdout: debug bluestore = 20 2021-08-21T14:59:37.556 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2021-08-21T14:59:37.557 INFO:teuthology.orchestra.run.smithi165.stdout: debug osd = 20 2021-08-21T14:59:37.557 INFO:teuthology.orchestra.run.smithi165.stdout: debug rocksdb = 10 2021-08-21T14:59:37.557 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd backfillfull_ratio = 0.85 2021-08-21T14:59:37.557 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd full ratio = 0.9 2021-08-21T14:59:37.558 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd nearfull ratio = 0.8 2021-08-21T14:59:37.558 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug verify cached snaps = True 2021-08-21T14:59:37.558 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug verify missing on start = True 2021-08-21T14:59:37.558 INFO:teuthology.orchestra.run.smithi165.stdout: osd failsafe full ratio = 0.95 2021-08-21T14:59:37.558 INFO:teuthology.orchestra.run.smithi165.stdout: osd objectstore = bluestore 2021-08-21T14:59:37.559 INFO:teuthology.orchestra.run.smithi165.stdout: osd op queue = debug_random 2021-08-21T14:59:37.559 INFO:teuthology.orchestra.run.smithi165.stdout: osd op queue cut off = debug_random 2021-08-21T14:59:37.559 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.559 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr] 2021-08-21T14:59:37.560 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2021-08-21T14:59:37.560 INFO:teuthology.orchestra.run.smithi165.stdout: debug mgr = 20 2021-08-21T14:59:37.560 INFO:teuthology.orchestra.run.smithi165.stdout: debug mon = 20 2021-08-21T14:59:37.560 INFO:teuthology.orchestra.run.smithi165.stdout: debug auth = 20 2021-08-21T14:59:37.560 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min pgs per osd = 4 2021-08-21T14:59:37.561 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min bytes per osd = 10 2021-08-21T14:59:37.561 INFO:teuthology.orchestra.run.smithi165.stdout: mgr/telemetry/nag = false 2021-08-21T14:59:37.561 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.561 INFO:teuthology.orchestra.run.smithi165.stdout:[mon] 2021-08-21T14:59:37.562 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2021-08-21T14:59:37.562 INFO:teuthology.orchestra.run.smithi165.stdout: debug mon = 20 2021-08-21T14:59:37.562 INFO:teuthology.orchestra.run.smithi165.stdout: debug paxos = 20 2021-08-21T14:59:37.562 INFO:teuthology.orchestra.run.smithi165.stdout: debug auth = 20 2021-08-21T14:59:37.562 INFO:teuthology.orchestra.run.smithi165.stdout: mon data avail warn = 5 2021-08-21T14:59:37.563 INFO:teuthology.orchestra.run.smithi165.stdout: mon mgr mkfs grace = 240 2021-08-21T14:59:37.563 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min pgs per osd = 4 2021-08-21T14:59:37.563 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd reporter subtree level = osd 2021-08-21T14:59:37.563 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd prime pg temp = true 2021-08-21T14:59:37.564 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min bytes per osd = 10 2021-08-21T14:59:37.564 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.564 INFO:teuthology.orchestra.run.smithi165.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-08-21T14:59:37.564 INFO:teuthology.orchestra.run.smithi165.stdout: auth mon ticket ttl = 660 # 11m 2021-08-21T14:59:37.564 INFO:teuthology.orchestra.run.smithi165.stdout: auth service ticket ttl = 240 # 4m 2021-08-21T14:59:37.565 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.565 INFO:teuthology.orchestra.run.smithi165.stdout: # don't complain about insecure global_id in the test suite 2021-08-21T14:59:37.565 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-08-21T14:59:37.565 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-08-21T14:59:37.565 INFO:teuthology.orchestra.run.smithi165.stdout: mon scrub interval = 300 2021-08-21T14:59:37.566 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T14:59:37.566 INFO:teuthology.orchestra.run.smithi165.stdout:[client] 2021-08-21T14:59:37.566 INFO:teuthology.orchestra.run.smithi165.stdout: rgw cache enabled = true 2021-08-21T14:59:37.566 INFO:teuthology.orchestra.run.smithi165.stdout: rgw enable ops log = true 2021-08-21T14:59:37.567 INFO:teuthology.orchestra.run.smithi165.stdout: rgw enable usage log = true 2021-08-21T14:59:37.567 INFO:teuthology.orchestra.run.smithi165.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-08-21T14:59:37.567 INFO:teuthology.orchestra.run.smithi165.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-08-21T14:59:37.567 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.a] 2021-08-21T14:59:37.568 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.b] 2021-08-21T14:59:37.568 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.c] 2021-08-21T14:59:37.569 INFO:tasks.ceph:Creating admin key on mon.a... 2021-08-21T14:59:37.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2021-08-21T14:59:37.617 INFO:tasks.ceph:Copying monmap to all nodes... 2021-08-21T14:59:37.617 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:59:37.618 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2021-08-21T14:59:37.624 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:59:37.624 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2021-08-21T14:59:37.672 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi165.front.sepia.ceph.com 2021-08-21T14:59:37.673 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:59:37.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.keyring 2021-08-21T14:59:37.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-08-21T14:59:37.700 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:59:37.701 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-08-21T14:59:37.705 INFO:tasks.ceph:Setting up mon nodes... 2021-08-21T14:59:37.705 INFO:tasks.ceph:Setting up mgr nodes... 2021-08-21T14:59:37.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2021-08-21T14:59:37.789 INFO:teuthology.orchestra.run.smithi165.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2021-08-21T14:59:37.792 INFO:tasks.ceph:Setting up mds nodes... 2021-08-21T14:59:37.792 INFO:tasks.ceph_client:Setting up client nodes... 2021-08-21T14:59:37.792 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-08-21T14:59:37.793 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi165.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2021-08-21T14:59:37.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2021-08-21T14:59:37.809 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-08-21T14:59:37.809 INFO:tasks.ceph:role: osd.0 2021-08-21T14:59:37.810 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi165.front.sepia.ceph.com 2021-08-21T14:59:37.810 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-08-21T14:59:38.336 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-08-21T14:59:38.336 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2021-08-21T14:59:38.336 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-08-21T14:59:38.337 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 2021-08-21T14:59:38.337 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-08-21T14:59:38.337 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2021-08-21T14:59:38.337 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-08-21T14:59:38.337 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-08-21T14:59:38.338 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-08-21T14:59:38.338 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-08-21T14:59:38.339 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2021-08-21T14:59:38.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2021-08-21T14:59:38.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2021-08-21T14:59:38.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-08-21T14:59:38.437 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: /sbin/restorecon: command not found 2021-08-21T14:59:38.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2021-08-21T14:59:38.449 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-08-21T14:59:38.450 INFO:tasks.ceph:role: osd.1 2021-08-21T14:59:38.450 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi165.front.sepia.ceph.com 2021-08-21T14:59:38.450 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-08-21T14:59:38.981 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-08-21T14:59:38.981 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2021-08-21T14:59:38.982 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-08-21T14:59:38.982 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 2021-08-21T14:59:38.982 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-08-21T14:59:38.983 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2021-08-21T14:59:38.983 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-08-21T14:59:38.983 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-08-21T14:59:38.983 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-08-21T14:59:38.983 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-08-21T14:59:38.984 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2021-08-21T14:59:38.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2021-08-21T14:59:39.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2021-08-21T14:59:39.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: /sbin/restorecon: command not found 2021-08-21T14:59:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-08-21T14:59:39.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2021-08-21T14:59:39.062 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-08-21T14:59:39.062 INFO:tasks.ceph:role: osd.2 2021-08-21T14:59:39.063 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi165.front.sepia.ceph.com 2021-08-21T14:59:39.063 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-08-21T14:59:39.535 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-08-21T14:59:39.535 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2021-08-21T14:59:39.535 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-08-21T14:59:39.536 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 2021-08-21T14:59:39.536 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-08-21T14:59:39.536 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2021-08-21T14:59:39.536 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-08-21T14:59:39.536 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-08-21T14:59:39.537 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-08-21T14:59:39.538 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-08-21T14:59:39.539 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2021-08-21T14:59:39.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2021-08-21T14:59:39.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2021-08-21T14:59:39.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-08-21T14:59:39.592 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: /sbin/restorecon: command not found 2021-08-21T14:59:39.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2021-08-21T14:59:39.651 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-08-21T14:59:39.652 INFO:tasks.ceph:role: osd.3 2021-08-21T14:59:39.652 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi165.front.sepia.ceph.com 2021-08-21T14:59:39.652 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2021-08-21T14:59:40.152 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2021-08-21T14:59:40.152 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2021-08-21T14:59:40.152 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-08-21T14:59:40.153 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 2021-08-21T14:59:40.153 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-08-21T14:59:40.153 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2021-08-21T14:59:40.153 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-08-21T14:59:40.153 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-08-21T14:59:40.154 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-08-21T14:59:40.154 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-08-21T14:59:40.155 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2021-08-21T14:59:40.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2021-08-21T14:59:40.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2021-08-21T14:59:40.218 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: /sbin/restorecon: command not found 2021-08-21T14:59:40.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-08-21T14:59:40.219 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-08-21T14:59:40.293 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:40.289+0000 7f70f5e15f40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2021-08-21T14:59:40.294 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:40.289+0000 7f70f5e15f40 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2021-08-21T14:59:40.295 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:40.289+0000 7f70f5e15f40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-08-21T14:59:40.295 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:40.289+0000 7f70f5e15f40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-08-21T14:59:40.295 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:40.289+0000 7f70f5e15f40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-08-21T14:59:40.296 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:40.293+0000 7f70f5e15f40 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2021-08-21T14:59:43.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-08-21T14:59:43.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-08-21T14:59:43.603 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:43.601+0000 7fc491260f40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2021-08-21T14:59:43.603 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:43.601+0000 7fc491260f40 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2021-08-21T14:59:43.604 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:43.601+0000 7fc491260f40 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-08-21T14:59:43.604 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:43.601+0000 7fc491260f40 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-08-21T14:59:43.604 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:43.601+0000 7fc491260f40 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-08-21T14:59:43.605 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:43.601+0000 7fc491260f40 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2021-08-21T14:59:46.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-08-21T14:59:46.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-08-21T14:59:46.661 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:46.657+0000 7f0506fe0f40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2021-08-21T14:59:46.662 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:46.657+0000 7f0506fe0f40 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2021-08-21T14:59:46.662 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:46.657+0000 7f0506fe0f40 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-08-21T14:59:46.663 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:46.657+0000 7f0506fe0f40 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-08-21T14:59:46.663 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:46.657+0000 7f0506fe0f40 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-08-21T14:59:46.663 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:46.661+0000 7f0506fe0f40 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2021-08-21T14:59:49.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-08-21T14:59:49.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-08-21T14:59:49.743 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:49.741+0000 7f43d2bb6f40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2021-08-21T14:59:49.743 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:49.741+0000 7f43d2bb6f40 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2021-08-21T14:59:49.744 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:49.741+0000 7f43d2bb6f40 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-08-21T14:59:49.744 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:49.741+0000 7f43d2bb6f40 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-08-21T14:59:49.744 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:49.741+0000 7f43d2bb6f40 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-08-21T14:59:49.745 INFO:teuthology.orchestra.run.smithi165.stderr:2021-08-21T14:59:49.741+0000 7f43d2bb6f40 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2021-08-21T14:59:52.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2021-08-21T14:59:52.731 INFO:tasks.ceph:Reading keys from all nodes... 2021-08-21T14:59:52.732 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:59:52.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2021-08-21T14:59:52.744 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:59:52.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2021-08-21T14:59:52.755 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:59:52.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2021-08-21T14:59:52.812 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:59:52.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2021-08-21T14:59:52.825 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T14:59:52.825 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2021-08-21T14:59:52.836 INFO:tasks.ceph:Adding keys to all mons... 2021-08-21T14:59:52.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo tee -a /etc/ceph/ceph.keyring 2021-08-21T14:59:52.892 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr.x] 2021-08-21T14:59:52.892 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQDZFCFhqcksLxAAjBIdMaRVadn9L4/tz2X0vQ== 2021-08-21T14:59:52.892 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.0] 2021-08-21T14:59:52.893 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQDcFCFh6nmJERAA3OY6a1YEvV/dmnpB2/KDtg== 2021-08-21T14:59:52.894 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.1] 2021-08-21T14:59:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQDfFCFh+48NJBAAykT2bQPi0SriHcVv+XvB7w== 2021-08-21T14:59:52.894 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.2] 2021-08-21T14:59:52.894 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQDiFCFhOMmIJxAA8qdAtAw8KWlhMft1epF2Zg== 2021-08-21T14:59:52.894 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.3] 2021-08-21T14:59:52.895 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQDlFCFhm9FlLBAAj01ZAhIv8oUI8Gfi95IlZg== 2021-08-21T14:59:52.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-08-21T14:59:52.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-08-21T14:59:52.983 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-08-21T14:59:53.070 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-08-21T14:59:53.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-08-21T14:59:53.153 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-08-21T14:59:53.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2021-08-21T14:59:53.210 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-08-21T14:59:53.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2021-08-21T14:59:53.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2021-08-21T14:59:53.334 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-08-21T14:59:53.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2021-08-21T14:59:53.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2021-08-21T14:59:53.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-08-21T14:59:53.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2021-08-21T14:59:53.533 DEBUG:teuthology.orchestra.run.smithi165:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-08-21T14:59:53.538 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2021-08-21T14:59:53.539 INFO:tasks.ceph.mon.a:Restarting daemon 2021-08-21T14:59:53.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-08-21T14:59:53.584 INFO:tasks.ceph.mon.a:Started 2021-08-21T14:59:53.584 INFO:tasks.ceph.mon.b:Restarting daemon 2021-08-21T14:59:53.585 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-08-21T14:59:53.589 INFO:tasks.ceph.mon.b:Started 2021-08-21T14:59:53.589 INFO:tasks.ceph.mon.c:Restarting daemon 2021-08-21T14:59:53.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-08-21T14:59:53.593 INFO:tasks.ceph.mon.c:Started 2021-08-21T14:59:53.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph config set mgr mgr_pool false --force 2021-08-21T15:00:00.022 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2021-08-21T15:00:00.023 INFO:tasks.ceph.mgr.x:Restarting daemon 2021-08-21T15:00:00.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2021-08-21T15:00:00.026 INFO:tasks.ceph.mgr.x:Started 2021-08-21T15:00:00.026 INFO:tasks.ceph:Setting crush tunables to default 2021-08-21T15:00:00.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd crush tunables default 2021-08-21T15:00:00.218 INFO:teuthology.orchestra.run.smithi165.stderr:adjusted tunables profile to default 2021-08-21T15:00:00.232 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2021-08-21T15:00:00.232 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T15:00:00.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2021-08-21T15:00:00.246 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T15:00:00.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2021-08-21T15:00:00.308 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T15:00:00.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2021-08-21T15:00:00.321 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2021-08-21T15:00:00.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2021-08-21T15:00:00.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new f6dc722a-fc2c-46bd-ade5-b15534b9209a 0 2021-08-21T15:00:00.577 INFO:teuthology.orchestra.run.smithi165.stdout:0 2021-08-21T15:00:00.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new a8ed204d-2f11-474e-85a1-d842c1c86929 1 2021-08-21T15:00:00.786 INFO:teuthology.orchestra.run.smithi165.stdout:1 2021-08-21T15:00:00.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new f47a338b-b210-48d7-9b2b-d2bc504ad371 2 2021-08-21T15:00:00.991 INFO:teuthology.orchestra.run.smithi165.stdout:2 2021-08-21T15:00:01.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 6ece6914-c949-456a-99db-3479cfa37199 3 2021-08-21T15:00:01.196 INFO:teuthology.orchestra.run.smithi165.stdout:3 2021-08-21T15:00:01.208 INFO:tasks.ceph.osd.0:Restarting daemon 2021-08-21T15:00:01.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2021-08-21T15:00:01.211 INFO:tasks.ceph.osd.0:Started 2021-08-21T15:00:01.211 INFO:tasks.ceph.osd.1:Restarting daemon 2021-08-21T15:00:01.212 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2021-08-21T15:00:01.215 INFO:tasks.ceph.osd.1:Started 2021-08-21T15:00:01.216 INFO:tasks.ceph.osd.2:Restarting daemon 2021-08-21T15:00:01.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2021-08-21T15:00:01.219 INFO:tasks.ceph.osd.2:Started 2021-08-21T15:00:01.220 INFO:tasks.ceph.osd.3:Restarting daemon 2021-08-21T15:00:01.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2021-08-21T15:00:01.223 INFO:tasks.ceph.osd.3:Started 2021-08-21T15:00:01.223 INFO:tasks.ceph:Waiting for OSDs to come up 2021-08-21T15:00:01.584 INFO:tasks.ceph.osd.2.smithi165.stderr:2021-08-21T15:00:01.581+0000 7f32220f4f40 -1 Falling back to public interface 2021-08-21T15:00:01.599 INFO:tasks.ceph.osd.1.smithi165.stderr:2021-08-21T15:00:01.597+0000 7f116332cf40 -1 Falling back to public interface 2021-08-21T15:00:01.626 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-08-21T15:00:01.811 INFO:tasks.ceph.osd.0.smithi165.stderr:2021-08-21T15:00:01.809+0000 7f4b77ddaf40 -1 Falling back to public interface 2021-08-21T15:00:01.813 INFO:teuthology.misc.health.smithi165.stdout: 2021-08-21T15:00:01.813 INFO:teuthology.misc.health.smithi165.stdout:{"epoch":6,"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","created":"2021-08-21T14:59:59.722740+0000","modified":"2021-08-21T15:00:01.186118+0000","last_up_change":"0.000000","last_in_change":"2021-08-21T15:00:01.186118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"f6dc722a-fc2c-46bd-ade5-b15534b9209a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a8ed204d-2f11-474e-85a1-d842c1c86929","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f47a338b-b210-48d7-9b2b-d2bc504ad371","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6ece6914-c949-456a-99db-3479cfa37199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-08-21T15:00:01.823 DEBUG:teuthology.misc:0 of 4 OSDs are up 2021-08-21T15:00:01.824 INFO:tasks.ceph.osd.3.smithi165.stderr:2021-08-21T15:00:01.821+0000 7f23d0db4f40 -1 Falling back to public interface 2021-08-21T15:00:05.348 INFO:tasks.ceph.osd.0.smithi165.stderr:2021-08-21T15:00:05.345+0000 7f4b77ddaf40 -1 osd.0 0 log_to_monitors {default=true} 2021-08-21T15:00:05.482 INFO:tasks.ceph.osd.1.smithi165.stderr:2021-08-21T15:00:05.477+0000 7f116332cf40 -1 osd.1 0 log_to_monitors {default=true} 2021-08-21T15:00:05.483 INFO:tasks.ceph.osd.2.smithi165.stderr:2021-08-21T15:00:05.481+0000 7f32220f4f40 -1 osd.2 0 log_to_monitors {default=true} 2021-08-21T15:00:05.589 INFO:tasks.ceph.osd.3.smithi165.stderr:2021-08-21T15:00:05.585+0000 7f23d0db4f40 -1 osd.3 0 log_to_monitors {default=true} 2021-08-21T15:00:06.469 INFO:tasks.ceph.osd.0.smithi165.stderr:2021-08-21T15:00:06.465+0000 7f4b7357c700 -1 osd.0 0 waiting for initial osdmap 2021-08-21T15:00:06.472 INFO:tasks.ceph.osd.0.smithi165.stderr:2021-08-21T15:00:06.469+0000 7f4b6eb24700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-08-21T15:00:07.479 INFO:tasks.ceph.osd.2.smithi165.stderr:2021-08-21T15:00:07.473+0000 7f321d896700 -1 osd.2 0 waiting for initial osdmap 2021-08-21T15:00:07.480 INFO:tasks.ceph.osd.1.smithi165.stderr:2021-08-21T15:00:07.473+0000 7f115eace700 -1 osd.1 0 waiting for initial osdmap 2021-08-21T15:00:07.480 INFO:tasks.ceph.osd.3.smithi165.stderr:2021-08-21T15:00:07.473+0000 7f23cdd7b700 -1 osd.3 0 waiting for initial osdmap 2021-08-21T15:00:07.481 INFO:tasks.ceph.osd.3.smithi165.stderr:2021-08-21T15:00:07.477+0000 7f23c7afe700 -1 osd.3 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-08-21T15:00:07.481 INFO:tasks.ceph.osd.1.smithi165.stderr:2021-08-21T15:00:07.477+0000 7f115a076700 -1 osd.1 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-08-21T15:00:07.482 INFO:tasks.ceph.osd.2.smithi165.stderr:2021-08-21T15:00:07.477+0000 7f3218e3e700 -1 osd.2 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-08-21T15:00:08.227 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-08-21T15:00:08.496 INFO:teuthology.misc.health.smithi165.stdout: 2021-08-21T15:00:08.496 INFO:teuthology.misc.health.smithi165.stdout:{"epoch":10,"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","created":"2021-08-21T14:59:59.722740+0000","modified":"2021-08-21T15:00:08.474222+0000","last_up_change":"2021-08-21T15:00:08.474222+0000","last_in_change":"2021-08-21T15:00:01.186118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"f6dc722a-fc2c-46bd-ade5-b15534b9209a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":16490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6809","nonce":16490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6811","nonce":16490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":16490}]},"public_addr":"172.21.15.165:6808/16490","cluster_addr":"172.21.15.165:6809/16490","heartbeat_back_addr":"172.21.15.165:6811/16490","heartbeat_front_addr":"172.21.15.165:6810/16490","state":["exists","up"]},{"osd":1,"uuid":"a8ed204d-2f11-474e-85a1-d842c1c86929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":16500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6805","nonce":16500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6807","nonce":16500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":16500}]},"public_addr":"172.21.15.165:6804/16500","cluster_addr":"172.21.15.165:6805/16500","heartbeat_back_addr":"172.21.15.165:6807/16500","heartbeat_front_addr":"172.21.15.165:6806/16500","state":["exists","up"]},{"osd":2,"uuid":"f47a338b-b210-48d7-9b2b-d2bc504ad371","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":16491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6801","nonce":16491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6803","nonce":16491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":16491}]},"public_addr":"172.21.15.165:6800/16491","cluster_addr":"172.21.15.165:6801/16491","heartbeat_back_addr":"172.21.15.165:6803/16491","heartbeat_front_addr":"172.21.15.165:6802/16491","state":["exists","up"]},{"osd":3,"uuid":"6ece6914-c949-456a-99db-3479cfa37199","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":16503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6813","nonce":16503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6815","nonce":16503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":16503}]},"public_addr":"172.21.15.165:6812/16503","cluster_addr":"172.21.15.165:6813/16503","heartbeat_back_addr":"172.21.15.165:6815/16503","heartbeat_front_addr":"172.21.15.165:6814/16503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.340679+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-08-21T15:00:08.509 DEBUG:teuthology.misc:4 of 4 OSDs are up 2021-08-21T15:00:08.510 INFO:tasks.ceph:Creating RBD pool 2021-08-21T15:00:08.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd pool create rbd 8 2021-08-21T15:00:09.510 INFO:teuthology.orchestra.run.smithi165.stderr:pool 'rbd' created 2021-08-21T15:00:09.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2021-08-21T15:00:10.518 INFO:teuthology.orchestra.run.smithi165.stderr:enabled application 'rbd' on pool 'rbd' 2021-08-21T15:00:10.529 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2021-08-21T15:00:10.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-08-21T15:00:10.530 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-08-21T15:00:10.789 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:10.790 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":12,"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","created":"2021-08-21T14:59:59.722740+0000","modified":"2021-08-21T15:00:10.496811+0000","last_up_change":"2021-08-21T15:00:08.474222+0000","last_in_change":"2021-08-21T15:00:01.186118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-08-21T15:00:08.821901+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f6dc722a-fc2c-46bd-ade5-b15534b9209a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":16490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6809","nonce":16490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6811","nonce":16490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":16490}]},"public_addr":"172.21.15.165:6808/16490","cluster_addr":"172.21.15.165:6809/16490","heartbeat_back_addr":"172.21.15.165:6811/16490","heartbeat_front_addr":"172.21.15.165:6810/16490","state":["exists","up"]},{"osd":1,"uuid":"a8ed204d-2f11-474e-85a1-d842c1c86929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":16500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6805","nonce":16500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6807","nonce":16500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":16500}]},"public_addr":"172.21.15.165:6804/16500","cluster_addr":"172.21.15.165:6805/16500","heartbeat_back_addr":"172.21.15.165:6807/16500","heartbeat_front_addr":"172.21.15.165:6806/16500","state":["exists","up"]},{"osd":2,"uuid":"f47a338b-b210-48d7-9b2b-d2bc504ad371","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":16491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6801","nonce":16491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6803","nonce":16491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":16491}]},"public_addr":"172.21.15.165:6800/16491","cluster_addr":"172.21.15.165:6801/16491","heartbeat_back_addr":"172.21.15.165:6803/16491","heartbeat_front_addr":"172.21.15.165:6802/16491","state":["exists","up"]},{"osd":3,"uuid":"6ece6914-c949-456a-99db-3479cfa37199","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":16503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6813","nonce":16503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6815","nonce":16503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":16503}]},"public_addr":"172.21.15.165:6812/16503","cluster_addr":"172.21.15.165:6813/16503","heartbeat_back_addr":"172.21.15.165:6815/16503","heartbeat_front_addr":"172.21.15.165:6814/16503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.340679+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.459612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.440552+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.634977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-08-21T15:00:10.802 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-08-21T15:00:08.821901+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-08-21T15:00:10.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-08-21T15:00:11.082 INFO:teuthology.orchestra.run.smithi165.stdout:pg_num: 8 2021-08-21T15:00:11.095 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-08-21T15:00:11.096 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2021-08-21T15:00:11.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2021-08-21T15:00:11.388 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:11.400 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":4,"active_gid":4110,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":16310}]},"active_addr":"172.21.15.165:6816/16310","active_change":"2021-08-21T15:00:03.429726+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":3182364592}]},{"addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":9415624}]},{"addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":1130811127}]}]}} 2021-08-21T15:00:11.402 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2021-08-21T15:00:11.403 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2021-08-21T15:00:11.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-08-21T15:00:11.674 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:11.674 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":13,"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","created":"2021-08-21T14:59:59.722740+0000","modified":"2021-08-21T15:00:11.513846+0000","last_up_change":"2021-08-21T15:00:08.474222+0000","last_in_change":"2021-08-21T15:00:01.186118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-08-21T15:00:08.821901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f6dc722a-fc2c-46bd-ade5-b15534b9209a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":16490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6809","nonce":16490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6811","nonce":16490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":16490}]},"public_addr":"172.21.15.165:6808/16490","cluster_addr":"172.21.15.165:6809/16490","heartbeat_back_addr":"172.21.15.165:6811/16490","heartbeat_front_addr":"172.21.15.165:6810/16490","state":["exists","up"]},{"osd":1,"uuid":"a8ed204d-2f11-474e-85a1-d842c1c86929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":16500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6805","nonce":16500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6807","nonce":16500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":16500}]},"public_addr":"172.21.15.165:6804/16500","cluster_addr":"172.21.15.165:6805/16500","heartbeat_back_addr":"172.21.15.165:6807/16500","heartbeat_front_addr":"172.21.15.165:6806/16500","state":["exists","up"]},{"osd":2,"uuid":"f47a338b-b210-48d7-9b2b-d2bc504ad371","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":16491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6801","nonce":16491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6803","nonce":16491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":16491}]},"public_addr":"172.21.15.165:6800/16491","cluster_addr":"172.21.15.165:6801/16491","heartbeat_back_addr":"172.21.15.165:6803/16491","heartbeat_front_addr":"172.21.15.165:6802/16491","state":["exists","up"]},{"osd":3,"uuid":"6ece6914-c949-456a-99db-3479cfa37199","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":16503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6813","nonce":16503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6815","nonce":16503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":16503}]},"public_addr":"172.21.15.165:6812/16503","cluster_addr":"172.21.15.165:6813/16503","heartbeat_back_addr":"172.21.15.165:6815/16503","heartbeat_front_addr":"172.21.15.165:6814/16503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.340679+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.459612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.440552+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.634977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-08-21T15:00:11.687 INFO:tasks.ceph.ceph_manager.ceph:all up! 2021-08-21T15:00:11.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-08-21T15:00:11.952 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:11.952 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":13,"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","created":"2021-08-21T14:59:59.722740+0000","modified":"2021-08-21T15:00:11.513846+0000","last_up_change":"2021-08-21T15:00:08.474222+0000","last_in_change":"2021-08-21T15:00:01.186118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-08-21T15:00:08.821901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f6dc722a-fc2c-46bd-ade5-b15534b9209a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":16490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6809","nonce":16490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6811","nonce":16490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":16490}]},"public_addr":"172.21.15.165:6808/16490","cluster_addr":"172.21.15.165:6809/16490","heartbeat_back_addr":"172.21.15.165:6811/16490","heartbeat_front_addr":"172.21.15.165:6810/16490","state":["exists","up"]},{"osd":1,"uuid":"a8ed204d-2f11-474e-85a1-d842c1c86929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":16500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6805","nonce":16500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6807","nonce":16500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":16500}]},"public_addr":"172.21.15.165:6804/16500","cluster_addr":"172.21.15.165:6805/16500","heartbeat_back_addr":"172.21.15.165:6807/16500","heartbeat_front_addr":"172.21.15.165:6806/16500","state":["exists","up"]},{"osd":2,"uuid":"f47a338b-b210-48d7-9b2b-d2bc504ad371","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":16491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6801","nonce":16491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6803","nonce":16491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":16491}]},"public_addr":"172.21.15.165:6800/16491","cluster_addr":"172.21.15.165:6801/16491","heartbeat_back_addr":"172.21.15.165:6803/16491","heartbeat_front_addr":"172.21.15.165:6802/16491","state":["exists","up"]},{"osd":3,"uuid":"6ece6914-c949-456a-99db-3479cfa37199","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":16503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6813","nonce":16503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6815","nonce":16503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":16503}]},"public_addr":"172.21.15.165:6812/16503","cluster_addr":"172.21.15.165:6813/16503","heartbeat_back_addr":"172.21.15.165:6815/16503","heartbeat_front_addr":"172.21.15.165:6814/16503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.340679+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.459612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.440552+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.634977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-08-21T15:00:11.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-08-21T15:00:12.114 INFO:teuthology.orchestra.run.smithi165.stdout:38654705667 2021-08-21T15:00:12.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-08-21T15:00:12.258 INFO:teuthology.orchestra.run.smithi165.stdout:42949672962 2021-08-21T15:00:12.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-08-21T15:00:12.403 INFO:teuthology.orchestra.run.smithi165.stdout:42949672963 2021-08-21T15:00:12.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2021-08-21T15:00:12.556 INFO:teuthology.orchestra.run.smithi165.stdout:42949672962 2021-08-21T15:00:12.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-08-21T15:00:12.822 INFO:teuthology.orchestra.run.smithi165.stdout:38654705666 2021-08-21T15:00:12.836 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.0 2021-08-21T15:00:13.837 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-08-21T15:00:14.122 INFO:teuthology.orchestra.run.smithi165.stdout:38654705667 2021-08-21T15:00:14.134 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.0 2021-08-21T15:00:14.135 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-08-21T15:00:14.399 INFO:teuthology.orchestra.run.smithi165.stdout:42949672962 2021-08-21T15:00:14.414 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672962 got 42949672962 for osd.1 2021-08-21T15:00:14.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-08-21T15:00:14.684 INFO:teuthology.orchestra.run.smithi165.stdout:42949672963 2021-08-21T15:00:14.699 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672963 got 42949672963 for osd.2 2021-08-21T15:00:14.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2021-08-21T15:00:14.965 INFO:teuthology.orchestra.run.smithi165.stdout:42949672962 2021-08-21T15:00:14.975 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672962 got 42949672962 for osd.3 2021-08-21T15:00:14.975 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2021-08-21T15:00:14.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:00:15.235 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:15.237 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:00:15.248 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2021-08-21T15:00:13.444388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":19784,"kb_used_data":424,"kb_used_omap":0,"kb_used_meta":19328,"kb_avail":377467576,"statfs":{"total":386547056640,"available":386526797824,"internally_reserved":0,"allocated":434176,"data_stored":100824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19791872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.514540+0000","last_change":"2021-08-21T15:00:10.514540+0000","last_active":"2021-08-21T15:00:10.514540+0000","last_peered":"2021-08-21T15:00:10.514540+0000","last_clean":"2021-08-21T15:00:10.514540+0000","last_became_active":"2021-08-21T15:00:10.514122+0000","last_became_peered":"2021-08-21T15:00:10.514122+0000","last_unstale":"2021-08-21T15:00:10.514540+0000","last_undegraded":"2021-08-21T15:00:10.514540+0000","last_fullsized":"2021-08-21T15:00:10.514540+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521864+0000","last_change":"2021-08-21T15:00:10.517834+0000","last_active":"2021-08-21T15:00:10.521864+0000","last_peered":"2021-08-21T15:00:10.521864+0000","last_clean":"2021-08-21T15:00:10.521864+0000","last_became_active":"2021-08-21T15:00:10.517549+0000","last_became_peered":"2021-08-21T15:00:10.517549+0000","last_unstale":"2021-08-21T15:00:10.521864+0000","last_undegraded":"2021-08-21T15:00:10.521864+0000","last_fullsized":"2021-08-21T15:00:10.521864+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521593+0000","last_change":"2021-08-21T15:00:10.516651+0000","last_active":"2021-08-21T15:00:10.521593+0000","last_peered":"2021-08-21T15:00:10.521593+0000","last_clean":"2021-08-21T15:00:10.521593+0000","last_became_active":"2021-08-21T15:00:10.516390+0000","last_became_peered":"2021-08-21T15:00:10.516390+0000","last_unstale":"2021-08-21T15:00:10.521593+0000","last_undegraded":"2021-08-21T15:00:10.521593+0000","last_fullsized":"2021-08-21T15:00:10.521593+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521979+0000","last_change":"2021-08-21T15:00:10.518266+0000","last_active":"2021-08-21T15:00:10.521979+0000","last_peered":"2021-08-21T15:00:10.521979+0000","last_clean":"2021-08-21T15:00:10.521979+0000","last_became_active":"2021-08-21T15:00:10.517033+0000","last_became_peered":"2021-08-21T15:00:10.517033+0000","last_unstale":"2021-08-21T15:00:10.521979+0000","last_undegraded":"2021-08-21T15:00:10.521979+0000","last_fullsized":"2021-08-21T15:00:10.521979+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.515843+0000","last_change":"2021-08-21T15:00:10.515843+0000","last_active":"2021-08-21T15:00:10.515843+0000","last_peered":"2021-08-21T15:00:10.515843+0000","last_clean":"2021-08-21T15:00:10.515843+0000","last_became_active":"2021-08-21T15:00:10.515545+0000","last_became_peered":"2021-08-21T15:00:10.515545+0000","last_unstale":"2021-08-21T15:00:10.515843+0000","last_undegraded":"2021-08-21T15:00:10.515843+0000","last_fullsized":"2021-08-21T15:00:10.515843+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.516644+0000","last_change":"2021-08-21T15:00:10.514314+0000","last_active":"2021-08-21T15:00:10.516644+0000","last_peered":"2021-08-21T15:00:10.516644+0000","last_clean":"2021-08-21T15:00:10.516644+0000","last_became_active":"2021-08-21T15:00:10.513913+0000","last_became_peered":"2021-08-21T15:00:10.513913+0000","last_unstale":"2021-08-21T15:00:10.516644+0000","last_undegraded":"2021-08-21T15:00:10.516644+0000","last_fullsized":"2021-08-21T15:00:10.516644+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521944+0000","last_change":"2021-08-21T15:00:10.518029+0000","last_active":"2021-08-21T15:00:10.521944+0000","last_peered":"2021-08-21T15:00:10.521944+0000","last_clean":"2021-08-21T15:00:10.521944+0000","last_became_active":"2021-08-21T15:00:10.516993+0000","last_became_peered":"2021-08-21T15:00:10.516993+0000","last_unstale":"2021-08-21T15:00:10.521944+0000","last_undegraded":"2021-08-21T15:00:10.521944+0000","last_fullsized":"2021-08-21T15:00:10.521944+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.515353+0000","last_change":"2021-08-21T15:00:10.515353+0000","last_active":"2021-08-21T15:00:10.515353+0000","last_peered":"2021-08-21T15:00:10.515353+0000","last_clean":"2021-08-21T15:00:10.515353+0000","last_became_active":"2021-08-21T15:00:10.515080+0000","last_became_peered":"2021-08-21T15:00:10.515080+0000","last_unstale":"2021-08-21T15:00:10.515353+0000","last_undegraded":"2021-08-21T15:00:10.515353+0000","last_fullsized":"2021-08-21T15:00:10.515353+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4984,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366856,"statfs":{"total":96636764160,"available":96631660544,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]}]},{"osd":2,"up_from":10,"seq":42949672963,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366928,"statfs":{"total":96636764160,"available":96631734272,"internally_reserved":0,"allocated":106496,"data_stored":24468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]}]},{"osd":1,"up_from":10,"seq":42949672962,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4912,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366928,"statfs":{"total":96636764160,"available":96631734272,"internally_reserved":0,"allocated":106496,"data_stored":24468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80500000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4976,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366864,"statfs":{"total":96636764160,"available":96631668736,"internally_reserved":0,"allocated":106496,"data_stored":24468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:00:15.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:00:15.518 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:15.520 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:00:15.531 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2021-08-21T15:00:15.444897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":19872,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":19392,"kb_avail":377467488,"statfs":{"total":386547056640,"available":386526707712,"internally_reserved":0,"allocated":458752,"data_stored":109680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19857408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.519323+0000","last_change":"2021-08-21T15:00:10.514540+0000","last_active":"2021-08-21T15:00:10.519323+0000","last_peered":"2021-08-21T15:00:10.519323+0000","last_clean":"2021-08-21T15:00:10.519323+0000","last_became_active":"2021-08-21T15:00:10.514122+0000","last_became_peered":"2021-08-21T15:00:10.514122+0000","last_unstale":"2021-08-21T15:00:10.519323+0000","last_undegraded":"2021-08-21T15:00:10.519323+0000","last_fullsized":"2021-08-21T15:00:10.519323+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521864+0000","last_change":"2021-08-21T15:00:10.517834+0000","last_active":"2021-08-21T15:00:10.521864+0000","last_peered":"2021-08-21T15:00:10.521864+0000","last_clean":"2021-08-21T15:00:10.521864+0000","last_became_active":"2021-08-21T15:00:10.517549+0000","last_became_peered":"2021-08-21T15:00:10.517549+0000","last_unstale":"2021-08-21T15:00:10.521864+0000","last_undegraded":"2021-08-21T15:00:10.521864+0000","last_fullsized":"2021-08-21T15:00:10.521864+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521593+0000","last_change":"2021-08-21T15:00:10.516651+0000","last_active":"2021-08-21T15:00:10.521593+0000","last_peered":"2021-08-21T15:00:10.521593+0000","last_clean":"2021-08-21T15:00:10.521593+0000","last_became_active":"2021-08-21T15:00:10.516390+0000","last_became_peered":"2021-08-21T15:00:10.516390+0000","last_unstale":"2021-08-21T15:00:10.521593+0000","last_undegraded":"2021-08-21T15:00:10.521593+0000","last_fullsized":"2021-08-21T15:00:10.521593+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521979+0000","last_change":"2021-08-21T15:00:10.518266+0000","last_active":"2021-08-21T15:00:10.521979+0000","last_peered":"2021-08-21T15:00:10.521979+0000","last_clean":"2021-08-21T15:00:10.521979+0000","last_became_active":"2021-08-21T15:00:10.517033+0000","last_became_peered":"2021-08-21T15:00:10.517033+0000","last_unstale":"2021-08-21T15:00:10.521979+0000","last_undegraded":"2021-08-21T15:00:10.521979+0000","last_fullsized":"2021-08-21T15:00:10.521979+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521878+0000","last_change":"2021-08-21T15:00:10.515843+0000","last_active":"2021-08-21T15:00:10.521878+0000","last_peered":"2021-08-21T15:00:10.521878+0000","last_clean":"2021-08-21T15:00:10.521878+0000","last_became_active":"2021-08-21T15:00:10.515545+0000","last_became_peered":"2021-08-21T15:00:10.515545+0000","last_unstale":"2021-08-21T15:00:10.521878+0000","last_undegraded":"2021-08-21T15:00:10.521878+0000","last_fullsized":"2021-08-21T15:00:10.521878+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.516644+0000","last_change":"2021-08-21T15:00:10.514314+0000","last_active":"2021-08-21T15:00:10.516644+0000","last_peered":"2021-08-21T15:00:10.516644+0000","last_clean":"2021-08-21T15:00:10.516644+0000","last_became_active":"2021-08-21T15:00:10.513913+0000","last_became_peered":"2021-08-21T15:00:10.513913+0000","last_unstale":"2021-08-21T15:00:10.516644+0000","last_undegraded":"2021-08-21T15:00:10.516644+0000","last_fullsized":"2021-08-21T15:00:10.516644+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521944+0000","last_change":"2021-08-21T15:00:10.518029+0000","last_active":"2021-08-21T15:00:10.521944+0000","last_peered":"2021-08-21T15:00:10.521944+0000","last_clean":"2021-08-21T15:00:10.521944+0000","last_became_active":"2021-08-21T15:00:10.516993+0000","last_became_peered":"2021-08-21T15:00:10.516993+0000","last_unstale":"2021-08-21T15:00:10.521944+0000","last_undegraded":"2021-08-21T15:00:10.521944+0000","last_fullsized":"2021-08-21T15:00:10.521944+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521793+0000","last_change":"2021-08-21T15:00:10.515353+0000","last_active":"2021-08-21T15:00:10.521793+0000","last_peered":"2021-08-21T15:00:10.521793+0000","last_clean":"2021-08-21T15:00:10.521793+0000","last_became_active":"2021-08-21T15:00:10.515080+0000","last_became_peered":"2021-08-21T15:00:10.515080+0000","last_unstale":"2021-08-21T15:00:10.521793+0000","last_undegraded":"2021-08-21T15:00:10.521793+0000","last_fullsized":"2021-08-21T15:00:10.521793+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672963,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4984,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366856,"statfs":{"total":96636764160,"available":96631660544,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":2,"up_from":10,"seq":42949672964,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366920,"statfs":{"total":96636764160,"available":96631726080,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":1,"up_from":10,"seq":42949672963,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4984,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366856,"statfs":{"total":96636764160,"available":96631660544,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4984,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366856,"statfs":{"total":96636764160,"available":96631660544,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:00:15.532 INFO:tasks.ceph.ceph_manager.ceph:clean! 2021-08-21T15:00:15.532 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-08-21T15:00:15.533 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2021-08-21T15:00:15.533 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-08-21T15:00:15.846 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:15.847 INFO:teuthology.orchestra.run.smithi165.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-08-21T15:00:15.860 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2021-08-21T15:00:15.860 INFO:teuthology.run_tasks:Running task osd_recovery.test_incomplete_pgs... 2021-08-21T15:00:15.872 INFO:tasks.osd_recovery:num_osds is 4 2021-08-21T15:00:15.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-08-21T15:00:16.138 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:16.138 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":13,"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","created":"2021-08-21T14:59:59.722740+0000","modified":"2021-08-21T15:00:11.513846+0000","last_up_change":"2021-08-21T15:00:08.474222+0000","last_in_change":"2021-08-21T15:00:01.186118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-08-21T15:00:08.821901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f6dc722a-fc2c-46bd-ade5-b15534b9209a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":16490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6809","nonce":16490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6811","nonce":16490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":16490}]},"public_addr":"172.21.15.165:6808/16490","cluster_addr":"172.21.15.165:6809/16490","heartbeat_back_addr":"172.21.15.165:6811/16490","heartbeat_front_addr":"172.21.15.165:6810/16490","state":["exists","up"]},{"osd":1,"uuid":"a8ed204d-2f11-474e-85a1-d842c1c86929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":16500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6805","nonce":16500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6807","nonce":16500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":16500}]},"public_addr":"172.21.15.165:6804/16500","cluster_addr":"172.21.15.165:6805/16500","heartbeat_back_addr":"172.21.15.165:6807/16500","heartbeat_front_addr":"172.21.15.165:6806/16500","state":["exists","up"]},{"osd":2,"uuid":"f47a338b-b210-48d7-9b2b-d2bc504ad371","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":16491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6801","nonce":16491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6803","nonce":16491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":16491}]},"public_addr":"172.21.15.165:6800/16491","cluster_addr":"172.21.15.165:6801/16491","heartbeat_back_addr":"172.21.15.165:6803/16491","heartbeat_front_addr":"172.21.15.165:6802/16491","state":["exists","up"]},{"osd":3,"uuid":"6ece6914-c949-456a-99db-3479cfa37199","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":16503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6813","nonce":16503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6815","nonce":16503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":16503}]},"public_addr":"172.21.15.165:6812/16503","cluster_addr":"172.21.15.165:6813/16503","heartbeat_back_addr":"172.21.15.165:6815/16503","heartbeat_front_addr":"172.21.15.165:6814/16503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.340679+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.459612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.440552+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-08-21T15:00:06.634977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-08-21T15:00:16.151 INFO:tasks.osd_recovery.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-08-21T15:00:08.821901+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-08-21T15:00:16.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-08-21T15:00:16.421 INFO:teuthology.orchestra.run.smithi165.stdout:pg_num: 8 2021-08-21T15:00:16.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2021-08-21T15:00:16.696 INFO:teuthology.orchestra.run.smithi165.stdout:epoch 13 2021-08-21T15:00:16.697 INFO:teuthology.orchestra.run.smithi165.stdout:fsid b82e8492-200f-4d81-953f-d1ffd38e4baa 2021-08-21T15:00:16.697 INFO:teuthology.orchestra.run.smithi165.stdout:created 2021-08-21T14:59:59.722740+0000 2021-08-21T15:00:16.697 INFO:teuthology.orchestra.run.smithi165.stdout:modified 2021-08-21T15:00:11.513846+0000 2021-08-21T15:00:16.697 INFO:teuthology.orchestra.run.smithi165.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2021-08-21T15:00:16.698 INFO:teuthology.orchestra.run.smithi165.stdout:crush_version 5 2021-08-21T15:00:16.698 INFO:teuthology.orchestra.run.smithi165.stdout:full_ratio 0.95 2021-08-21T15:00:16.698 INFO:teuthology.orchestra.run.smithi165.stdout:backfillfull_ratio 0.9 2021-08-21T15:00:16.698 INFO:teuthology.orchestra.run.smithi165.stdout:nearfull_ratio 0.85 2021-08-21T15:00:16.699 INFO:teuthology.orchestra.run.smithi165.stdout:require_min_compat_client luminous 2021-08-21T15:00:16.699 INFO:teuthology.orchestra.run.smithi165.stdout:min_compat_client jewel 2021-08-21T15:00:16.699 INFO:teuthology.orchestra.run.smithi165.stdout:require_osd_release quincy 2021-08-21T15:00:16.699 INFO:teuthology.orchestra.run.smithi165.stdout:stretch_mode_enabled false 2021-08-21T15:00:16.700 INFO:teuthology.orchestra.run.smithi165.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 13 flags hashpspool stripe_width 0 application rbd 2021-08-21T15:00:16.700 INFO:teuthology.orchestra.run.smithi165.stdout:max_osd 4 2021-08-21T15:00:16.700 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.165:6808/16490 v2:172.21.15.165:6809/16490 exists,up f6dc722a-fc2c-46bd-ade5-b15534b9209a 2021-08-21T15:00:16.700 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.165:6804/16500 v2:172.21.15.165:6805/16500 exists,up a8ed204d-2f11-474e-85a1-d842c1c86929 2021-08-21T15:00:16.701 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.165:6800/16491 v2:172.21.15.165:6801/16491 exists,up f47a338b-b210-48d7-9b2b-d2bc504ad371 2021-08-21T15:00:16.701 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.165:6812/16503 v2:172.21.15.165:6813/16503 exists,up 6ece6914-c949-456a-99db-3479cfa37199 2021-08-21T15:00:16.710 INFO:tasks.osd_recovery.ceph_manager:['osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.165:6808/16490 v2:172.21.15.165:6809/16490 exists,up f6dc722a-fc2c-46bd-ade5-b15534b9209a', 'osd.1 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.165:6804/16500 v2:172.21.15.165:6805/16500 exists,up a8ed204d-2f11-474e-85a1-d842c1c86929', 'osd.2 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.165:6800/16491 v2:172.21.15.165:6801/16491 exists,up f47a338b-b210-48d7-9b2b-d2bc504ad371', 'osd.3 up in weight 1 up_from 10 up_thru 11 down_at 0 last_clean_interval [0,0) v2:172.21.15.165:6812/16503 v2:172.21.15.165:6813/16503 exists,up 6ece6914-c949-456a-99db-3479cfa37199'] 2021-08-21T15:00:16.711 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-08-21T15:00:16.851 INFO:teuthology.orchestra.run.smithi165.stdout:38654705669 2021-08-21T15:00:16.852 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-08-21T15:00:16.995 INFO:teuthology.orchestra.run.smithi165.stdout:42949672964 2021-08-21T15:00:16.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-08-21T15:00:17.141 INFO:teuthology.orchestra.run.smithi165.stdout:42949672965 2021-08-21T15:00:17.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2021-08-21T15:00:17.290 INFO:teuthology.orchestra.run.smithi165.stdout:42949672964 2021-08-21T15:00:17.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-08-21T15:00:17.559 INFO:teuthology.orchestra.run.smithi165.stdout:38654705668 2021-08-21T15:00:17.572 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705669 got 38654705668 for osd.0 2021-08-21T15:00:18.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-08-21T15:00:18.854 INFO:teuthology.orchestra.run.smithi165.stdout:38654705669 2021-08-21T15:00:18.873 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705669 got 38654705669 for osd.0 2021-08-21T15:00:18.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-08-21T15:00:19.141 INFO:teuthology.orchestra.run.smithi165.stdout:42949672964 2021-08-21T15:00:19.275 INFO:tasks.osd_recovery.ceph_manager:need seq 42949672964 got 42949672964 for osd.1 2021-08-21T15:00:19.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-08-21T15:00:19.550 INFO:teuthology.orchestra.run.smithi165.stdout:42949672965 2021-08-21T15:00:19.563 INFO:tasks.osd_recovery.ceph_manager:need seq 42949672965 got 42949672965 for osd.2 2021-08-21T15:00:19.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2021-08-21T15:00:19.830 INFO:teuthology.orchestra.run.smithi165.stdout:42949672964 2021-08-21T15:00:19.843 INFO:tasks.osd_recovery.ceph_manager:need seq 42949672964 got 42949672964 for osd.3 2021-08-21T15:00:19.844 INFO:tasks.osd_recovery.ceph_manager:waiting for clean 2021-08-21T15:00:19.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:00:20.099 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:20.101 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:00:20.111 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2021-08-21T15:00:19.445914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":19872,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":19392,"kb_avail":377467488,"statfs":{"total":386547056640,"available":386526707712,"internally_reserved":0,"allocated":458752,"data_stored":109680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19857408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.519323+0000","last_change":"2021-08-21T15:00:10.514540+0000","last_active":"2021-08-21T15:00:10.519323+0000","last_peered":"2021-08-21T15:00:10.519323+0000","last_clean":"2021-08-21T15:00:10.519323+0000","last_became_active":"2021-08-21T15:00:10.514122+0000","last_became_peered":"2021-08-21T15:00:10.514122+0000","last_unstale":"2021-08-21T15:00:10.519323+0000","last_undegraded":"2021-08-21T15:00:10.519323+0000","last_fullsized":"2021-08-21T15:00:10.519323+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521864+0000","last_change":"2021-08-21T15:00:10.517834+0000","last_active":"2021-08-21T15:00:10.521864+0000","last_peered":"2021-08-21T15:00:10.521864+0000","last_clean":"2021-08-21T15:00:10.521864+0000","last_became_active":"2021-08-21T15:00:10.517549+0000","last_became_peered":"2021-08-21T15:00:10.517549+0000","last_unstale":"2021-08-21T15:00:10.521864+0000","last_undegraded":"2021-08-21T15:00:10.521864+0000","last_fullsized":"2021-08-21T15:00:10.521864+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521593+0000","last_change":"2021-08-21T15:00:10.516651+0000","last_active":"2021-08-21T15:00:10.521593+0000","last_peered":"2021-08-21T15:00:10.521593+0000","last_clean":"2021-08-21T15:00:10.521593+0000","last_became_active":"2021-08-21T15:00:10.516390+0000","last_became_peered":"2021-08-21T15:00:10.516390+0000","last_unstale":"2021-08-21T15:00:10.521593+0000","last_undegraded":"2021-08-21T15:00:10.521593+0000","last_fullsized":"2021-08-21T15:00:10.521593+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521979+0000","last_change":"2021-08-21T15:00:10.518266+0000","last_active":"2021-08-21T15:00:10.521979+0000","last_peered":"2021-08-21T15:00:10.521979+0000","last_clean":"2021-08-21T15:00:10.521979+0000","last_became_active":"2021-08-21T15:00:10.517033+0000","last_became_peered":"2021-08-21T15:00:10.517033+0000","last_unstale":"2021-08-21T15:00:10.521979+0000","last_undegraded":"2021-08-21T15:00:10.521979+0000","last_fullsized":"2021-08-21T15:00:10.521979+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521878+0000","last_change":"2021-08-21T15:00:10.515843+0000","last_active":"2021-08-21T15:00:10.521878+0000","last_peered":"2021-08-21T15:00:10.521878+0000","last_clean":"2021-08-21T15:00:10.521878+0000","last_became_active":"2021-08-21T15:00:10.515545+0000","last_became_peered":"2021-08-21T15:00:10.515545+0000","last_unstale":"2021-08-21T15:00:10.521878+0000","last_undegraded":"2021-08-21T15:00:10.521878+0000","last_fullsized":"2021-08-21T15:00:10.521878+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.516644+0000","last_change":"2021-08-21T15:00:10.514314+0000","last_active":"2021-08-21T15:00:10.516644+0000","last_peered":"2021-08-21T15:00:10.516644+0000","last_clean":"2021-08-21T15:00:10.516644+0000","last_became_active":"2021-08-21T15:00:10.513913+0000","last_became_peered":"2021-08-21T15:00:10.513913+0000","last_unstale":"2021-08-21T15:00:10.516644+0000","last_undegraded":"2021-08-21T15:00:10.516644+0000","last_fullsized":"2021-08-21T15:00:10.516644+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521944+0000","last_change":"2021-08-21T15:00:10.518029+0000","last_active":"2021-08-21T15:00:10.521944+0000","last_peered":"2021-08-21T15:00:10.521944+0000","last_clean":"2021-08-21T15:00:10.521944+0000","last_became_active":"2021-08-21T15:00:10.516993+0000","last_became_peered":"2021-08-21T15:00:10.516993+0000","last_unstale":"2021-08-21T15:00:10.521944+0000","last_undegraded":"2021-08-21T15:00:10.521944+0000","last_fullsized":"2021-08-21T15:00:10.521944+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521793+0000","last_change":"2021-08-21T15:00:10.515353+0000","last_active":"2021-08-21T15:00:10.521793+0000","last_peered":"2021-08-21T15:00:10.521793+0000","last_clean":"2021-08-21T15:00:10.521793+0000","last_became_active":"2021-08-21T15:00:10.515080+0000","last_became_peered":"2021-08-21T15:00:10.515080+0000","last_unstale":"2021-08-21T15:00:10.521793+0000","last_undegraded":"2021-08-21T15:00:10.521793+0000","last_fullsized":"2021-08-21T15:00:10.521793+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672964,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4984,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366856,"statfs":{"total":96636764160,"available":96631660544,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":2,"up_from":10,"seq":42949672965,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366920,"statfs":{"total":96636764160,"available":96631726080,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]}]},{"osd":1,"up_from":10,"seq":42949672964,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4984,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366856,"statfs":{"total":96636764160,"available":96631660544,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4984,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366856,"statfs":{"total":96636764160,"available":96631660544,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:00:20.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:00:20.378 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:20.380 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:00:20.391 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2021-08-21T15:00:19.445914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":19872,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":19392,"kb_avail":377467488,"statfs":{"total":386547056640,"available":386526707712,"internally_reserved":0,"allocated":458752,"data_stored":109680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19857408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.519323+0000","last_change":"2021-08-21T15:00:10.514540+0000","last_active":"2021-08-21T15:00:10.519323+0000","last_peered":"2021-08-21T15:00:10.519323+0000","last_clean":"2021-08-21T15:00:10.519323+0000","last_became_active":"2021-08-21T15:00:10.514122+0000","last_became_peered":"2021-08-21T15:00:10.514122+0000","last_unstale":"2021-08-21T15:00:10.519323+0000","last_undegraded":"2021-08-21T15:00:10.519323+0000","last_fullsized":"2021-08-21T15:00:10.519323+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521864+0000","last_change":"2021-08-21T15:00:10.517834+0000","last_active":"2021-08-21T15:00:10.521864+0000","last_peered":"2021-08-21T15:00:10.521864+0000","last_clean":"2021-08-21T15:00:10.521864+0000","last_became_active":"2021-08-21T15:00:10.517549+0000","last_became_peered":"2021-08-21T15:00:10.517549+0000","last_unstale":"2021-08-21T15:00:10.521864+0000","last_undegraded":"2021-08-21T15:00:10.521864+0000","last_fullsized":"2021-08-21T15:00:10.521864+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521593+0000","last_change":"2021-08-21T15:00:10.516651+0000","last_active":"2021-08-21T15:00:10.521593+0000","last_peered":"2021-08-21T15:00:10.521593+0000","last_clean":"2021-08-21T15:00:10.521593+0000","last_became_active":"2021-08-21T15:00:10.516390+0000","last_became_peered":"2021-08-21T15:00:10.516390+0000","last_unstale":"2021-08-21T15:00:10.521593+0000","last_undegraded":"2021-08-21T15:00:10.521593+0000","last_fullsized":"2021-08-21T15:00:10.521593+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521979+0000","last_change":"2021-08-21T15:00:10.518266+0000","last_active":"2021-08-21T15:00:10.521979+0000","last_peered":"2021-08-21T15:00:10.521979+0000","last_clean":"2021-08-21T15:00:10.521979+0000","last_became_active":"2021-08-21T15:00:10.517033+0000","last_became_peered":"2021-08-21T15:00:10.517033+0000","last_unstale":"2021-08-21T15:00:10.521979+0000","last_undegraded":"2021-08-21T15:00:10.521979+0000","last_fullsized":"2021-08-21T15:00:10.521979+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521878+0000","last_change":"2021-08-21T15:00:10.515843+0000","last_active":"2021-08-21T15:00:10.521878+0000","last_peered":"2021-08-21T15:00:10.521878+0000","last_clean":"2021-08-21T15:00:10.521878+0000","last_became_active":"2021-08-21T15:00:10.515545+0000","last_became_peered":"2021-08-21T15:00:10.515545+0000","last_unstale":"2021-08-21T15:00:10.521878+0000","last_undegraded":"2021-08-21T15:00:10.521878+0000","last_fullsized":"2021-08-21T15:00:10.521878+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.516644+0000","last_change":"2021-08-21T15:00:10.514314+0000","last_active":"2021-08-21T15:00:10.516644+0000","last_peered":"2021-08-21T15:00:10.516644+0000","last_clean":"2021-08-21T15:00:10.516644+0000","last_became_active":"2021-08-21T15:00:10.513913+0000","last_became_peered":"2021-08-21T15:00:10.513913+0000","last_unstale":"2021-08-21T15:00:10.516644+0000","last_undegraded":"2021-08-21T15:00:10.516644+0000","last_fullsized":"2021-08-21T15:00:10.516644+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521944+0000","last_change":"2021-08-21T15:00:10.518029+0000","last_active":"2021-08-21T15:00:10.521944+0000","last_peered":"2021-08-21T15:00:10.521944+0000","last_clean":"2021-08-21T15:00:10.521944+0000","last_became_active":"2021-08-21T15:00:10.516993+0000","last_became_peered":"2021-08-21T15:00:10.516993+0000","last_unstale":"2021-08-21T15:00:10.521944+0000","last_undegraded":"2021-08-21T15:00:10.521944+0000","last_fullsized":"2021-08-21T15:00:10.521944+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-08-21T15:00:10.521793+0000","last_change":"2021-08-21T15:00:10.515353+0000","last_active":"2021-08-21T15:00:10.521793+0000","last_peered":"2021-08-21T15:00:10.521793+0000","last_clean":"2021-08-21T15:00:10.521793+0000","last_became_active":"2021-08-21T15:00:10.515080+0000","last_became_peered":"2021-08-21T15:00:10.515080+0000","last_unstale":"2021-08-21T15:00:10.521793+0000","last_undegraded":"2021-08-21T15:00:10.521793+0000","last_fullsized":"2021-08-21T15:00:10.521793+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672964,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4984,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366856,"statfs":{"total":96636764160,"available":96631660544,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":2,"up_from":10,"seq":42949672965,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366920,"statfs":{"total":96636764160,"available":96631726080,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]}]},{"osd":1,"up_from":10,"seq":42949672964,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4984,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366856,"statfs":{"total":96636764160,"available":96631660544,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4984,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366856,"statfs":{"total":96636764160,"available":96631660544,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:00:20.392 INFO:tasks.osd_recovery.ceph_manager:clean! 2021-08-21T15:00:20.392 INFO:tasks.osd_recovery:Testing incomplete pgs... 2021-08-21T15:00:20.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set osd_recovery_delay_start 1000 2021-08-21T15:00:20.496 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2021-08-21T15:00:20.497 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "osd_recovery_delay_start = '1000.000000' " 2021-08-21T15:00:20.497 INFO:teuthology.orchestra.run.smithi165.stdout:} 2021-08-21T15:00:20.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set osd_recovery_delay_start 1000 2021-08-21T15:00:20.606 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2021-08-21T15:00:20.607 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "osd_recovery_delay_start = '1000.000000' " 2021-08-21T15:00:20.607 INFO:teuthology.orchestra.run.smithi165.stdout:} 2021-08-21T15:00:20.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set osd_recovery_delay_start 1000 2021-08-21T15:00:20.715 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2021-08-21T15:00:20.716 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "osd_recovery_delay_start = '1000.000000' " 2021-08-21T15:00:20.716 INFO:teuthology.orchestra.run.smithi165.stdout:} 2021-08-21T15:00:20.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set osd_recovery_delay_start 1000 2021-08-21T15:00:20.821 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2021-08-21T15:00:20.821 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "osd_recovery_delay_start = '1000.000000' " 2021-08-21T15:00:20.822 INFO:teuthology.orchestra.run.smithi165.stdout:} 2021-08-21T15:00:20.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 1 2021-08-21T15:00:21.111 INFO:tasks.ceph.mon.a.smithi165.stderr:2021-08-21T15:00:21.109+0000 7f927d985700 -1 mon.a@0(leader).osd e13 definitely_dead 0 2021-08-21T15:00:21.745 INFO:teuthology.orchestra.run.smithi165.stderr:marked out osd.0. marked out osd.1. 2021-08-21T15:00:21.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-08-21T15:00:21.895 INFO:teuthology.orchestra.run.smithi165.stdout:38654705671 2021-08-21T15:00:21.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-08-21T15:00:22.045 INFO:teuthology.orchestra.run.smithi165.stdout:42949672966 2021-08-21T15:00:22.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-08-21T15:00:22.196 INFO:teuthology.orchestra.run.smithi165.stdout:42949672967 2021-08-21T15:00:22.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2021-08-21T15:00:22.346 INFO:teuthology.orchestra.run.smithi165.stdout:42949672966 2021-08-21T15:00:22.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-08-21T15:00:22.615 INFO:teuthology.orchestra.run.smithi165.stdout:42949672966 2021-08-21T15:00:22.632 INFO:tasks.osd_recovery.ceph_manager:need seq 42949672967 got 42949672966 for osd.2 2021-08-21T15:00:23.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-08-21T15:00:23.908 INFO:teuthology.orchestra.run.smithi165.stdout:42949672967 2021-08-21T15:00:23.921 INFO:tasks.osd_recovery.ceph_manager:need seq 42949672967 got 42949672967 for osd.2 2021-08-21T15:00:23.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2021-08-21T15:00:24.182 INFO:teuthology.orchestra.run.smithi165.stdout:42949672966 2021-08-21T15:00:24.207 INFO:tasks.osd_recovery.ceph_manager:need seq 42949672966 got 42949672966 for osd.3 2021-08-21T15:00:24.208 INFO:tasks.osd_recovery.ceph_manager:waiting for clean 2021-08-21T15:00:24.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:00:24.476 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:24.480 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:00:24.496 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2021-08-21T15:00:23.446989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":9904,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":9664,"kb_avail":188733776,"statfs":{"total":193273528320,"available":193263386624,"internally_reserved":0,"allocated":229376,"data_stored":54840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9895936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.742162+0000","last_change":"2021-08-21T15:00:21.742426+0000","last_active":"2021-08-21T15:00:09.483517+0000","last_peered":"2021-08-21T15:00:09.483517+0000","last_clean":"2021-08-21T15:00:09.483517+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2021-08-21T15:00:21.742162+0000","last_undegraded":"2021-08-21T15:00:21.742162+0000","last_fullsized":"2021-08-21T15:00:21.742162+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.740687+0000","last_change":"2021-08-21T15:00:21.743257+0000","last_active":"2021-08-21T15:00:21.740687+0000","last_peered":"2021-08-21T15:00:11.535600+0000","last_clean":"2021-08-21T15:00:11.535600+0000","last_became_active":"2021-08-21T15:00:10.517549+0000","last_became_peered":"2021-08-21T15:00:10.517549+0000","last_unstale":"2021-08-21T15:00:21.740687+0000","last_undegraded":"2021-08-21T15:00:21.740687+0000","last_fullsized":"2021-08-21T15:00:21.740687+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2021-08-21T15:00:11.891325+0000","last_change":"2021-08-21T15:00:10.516651+0000","last_active":"2021-08-21T15:00:11.891325+0000","last_peered":"2021-08-21T15:00:11.891325+0000","last_clean":"2021-08-21T15:00:11.891325+0000","last_became_active":"2021-08-21T15:00:10.516390+0000","last_became_peered":"2021-08-21T15:00:10.516390+0000","last_unstale":"2021-08-21T15:00:11.891325+0000","last_undegraded":"2021-08-21T15:00:11.891325+0000","last_fullsized":"2021-08-21T15:00:11.891325+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.740663+0000","last_change":"2021-08-21T15:00:21.741173+0000","last_active":"2021-08-21T15:00:21.740663+0000","last_peered":"2021-08-21T15:00:11.535607+0000","last_clean":"2021-08-21T15:00:11.535607+0000","last_became_active":"2021-08-21T15:00:10.517033+0000","last_became_peered":"2021-08-21T15:00:10.517033+0000","last_unstale":"2021-08-21T15:00:21.740663+0000","last_undegraded":"2021-08-21T15:00:21.740663+0000","last_fullsized":"2021-08-21T15:00:21.740663+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.734728+0000","last_change":"2021-08-21T15:00:21.735069+0000","last_active":"2021-08-21T15:00:09.483517+0000","last_peered":"2021-08-21T15:00:09.483517+0000","last_clean":"2021-08-21T15:00:09.483517+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2021-08-21T15:00:21.734728+0000","last_undegraded":"2021-08-21T15:00:21.734728+0000","last_fullsized":"2021-08-21T15:00:21.734728+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.740520+0000","last_change":"2021-08-21T15:00:21.740520+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2021-08-21T15:00:21.740520+0000","last_undegraded":"2021-08-21T15:00:21.740520+0000","last_fullsized":"2021-08-21T15:00:21.740520+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.742166+0000","last_change":"2021-08-21T15:00:21.742352+0000","last_active":"2021-08-21T15:00:21.742166+0000","last_peered":"2021-08-21T15:00:11.535480+0000","last_clean":"2021-08-21T15:00:11.535480+0000","last_became_active":"2021-08-21T15:00:10.516993+0000","last_became_peered":"2021-08-21T15:00:10.516993+0000","last_unstale":"2021-08-21T15:00:21.742166+0000","last_undegraded":"2021-08-21T15:00:21.742166+0000","last_fullsized":"2021-08-21T15:00:21.742166+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":1,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.738352+0000","last_change":"2021-08-21T15:00:21.738352+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2021-08-21T15:00:21.738352+0000","last_undegraded":"2021-08-21T15:00:21.738352+0000","last_fullsized":"2021-08-21T15:00:21.738352+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672966,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4984,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366856,"statfs":{"total":96636764160,"available":96631660544,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":2,"up_from":10,"seq":42949672967,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366920,"statfs":{"total":96636764160,"available":96631726080,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]}]},{"osd":1,"up_from":0,"seq":42949672966,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705671,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:00:24.501 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:00:24.769 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:24.771 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:00:24.781 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2021-08-21T15:00:23.446989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":9904,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":9664,"kb_avail":188733776,"statfs":{"total":193273528320,"available":193263386624,"internally_reserved":0,"allocated":229376,"data_stored":54840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9895936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.742162+0000","last_change":"2021-08-21T15:00:21.742426+0000","last_active":"2021-08-21T15:00:09.483517+0000","last_peered":"2021-08-21T15:00:09.483517+0000","last_clean":"2021-08-21T15:00:09.483517+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2021-08-21T15:00:21.742162+0000","last_undegraded":"2021-08-21T15:00:21.742162+0000","last_fullsized":"2021-08-21T15:00:21.742162+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.740687+0000","last_change":"2021-08-21T15:00:21.743257+0000","last_active":"2021-08-21T15:00:21.740687+0000","last_peered":"2021-08-21T15:00:11.535600+0000","last_clean":"2021-08-21T15:00:11.535600+0000","last_became_active":"2021-08-21T15:00:10.517549+0000","last_became_peered":"2021-08-21T15:00:10.517549+0000","last_unstale":"2021-08-21T15:00:21.740687+0000","last_undegraded":"2021-08-21T15:00:21.740687+0000","last_fullsized":"2021-08-21T15:00:21.740687+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2021-08-21T15:00:11.891325+0000","last_change":"2021-08-21T15:00:10.516651+0000","last_active":"2021-08-21T15:00:11.891325+0000","last_peered":"2021-08-21T15:00:11.891325+0000","last_clean":"2021-08-21T15:00:11.891325+0000","last_became_active":"2021-08-21T15:00:10.516390+0000","last_became_peered":"2021-08-21T15:00:10.516390+0000","last_unstale":"2021-08-21T15:00:11.891325+0000","last_undegraded":"2021-08-21T15:00:11.891325+0000","last_fullsized":"2021-08-21T15:00:11.891325+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.740663+0000","last_change":"2021-08-21T15:00:21.741173+0000","last_active":"2021-08-21T15:00:21.740663+0000","last_peered":"2021-08-21T15:00:11.535607+0000","last_clean":"2021-08-21T15:00:11.535607+0000","last_became_active":"2021-08-21T15:00:10.517033+0000","last_became_peered":"2021-08-21T15:00:10.517033+0000","last_unstale":"2021-08-21T15:00:21.740663+0000","last_undegraded":"2021-08-21T15:00:21.740663+0000","last_fullsized":"2021-08-21T15:00:21.740663+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.734728+0000","last_change":"2021-08-21T15:00:21.735069+0000","last_active":"2021-08-21T15:00:09.483517+0000","last_peered":"2021-08-21T15:00:09.483517+0000","last_clean":"2021-08-21T15:00:09.483517+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2021-08-21T15:00:21.734728+0000","last_undegraded":"2021-08-21T15:00:21.734728+0000","last_fullsized":"2021-08-21T15:00:21.734728+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.740520+0000","last_change":"2021-08-21T15:00:21.740520+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2021-08-21T15:00:21.740520+0000","last_undegraded":"2021-08-21T15:00:21.740520+0000","last_fullsized":"2021-08-21T15:00:21.740520+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.742166+0000","last_change":"2021-08-21T15:00:21.742352+0000","last_active":"2021-08-21T15:00:21.742166+0000","last_peered":"2021-08-21T15:00:11.535480+0000","last_clean":"2021-08-21T15:00:11.535480+0000","last_became_active":"2021-08-21T15:00:10.516993+0000","last_became_peered":"2021-08-21T15:00:10.516993+0000","last_unstale":"2021-08-21T15:00:21.742166+0000","last_undegraded":"2021-08-21T15:00:21.742166+0000","last_fullsized":"2021-08-21T15:00:21.742166+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":1,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.738352+0000","last_change":"2021-08-21T15:00:21.738352+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2021-08-21T15:00:21.738352+0000","last_undegraded":"2021-08-21T15:00:21.738352+0000","last_fullsized":"2021-08-21T15:00:21.738352+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672966,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4984,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366856,"statfs":{"total":96636764160,"available":96631660544,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":2,"up_from":10,"seq":42949672967,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366920,"statfs":{"total":96636764160,"available":96631726080,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]}]},{"osd":1,"up_from":0,"seq":42949672966,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705671,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:00:24.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:00:25.090 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:25.090 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":25,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":15,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558008,"num_in_osds":2,"osd_in_since":1629558021,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"peering","count":7},{"state_name":"active+clean","count":1}],"num_pgs":8,"num_pools":1,"num_objects":0,"data_bytes":0,"bytes_used":10141696,"bytes_avail":193263386624,"bytes_total":193273528320,"inactive_pgs_ratio":0.875},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2021-08-21T14:59:59.724199+0000","services":{}},"progress_events":{}} 2021-08-21T15:00:25.103 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2021-08-21T15:00:25.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:00:25.371 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:25.374 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:00:25.388 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2021-08-21T15:00:23.446989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":9904,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":9664,"kb_avail":188733776,"statfs":{"total":193273528320,"available":193263386624,"internally_reserved":0,"allocated":229376,"data_stored":54840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9895936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.742162+0000","last_change":"2021-08-21T15:00:21.742426+0000","last_active":"2021-08-21T15:00:09.483517+0000","last_peered":"2021-08-21T15:00:09.483517+0000","last_clean":"2021-08-21T15:00:09.483517+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2021-08-21T15:00:21.742162+0000","last_undegraded":"2021-08-21T15:00:21.742162+0000","last_fullsized":"2021-08-21T15:00:21.742162+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.740687+0000","last_change":"2021-08-21T15:00:21.743257+0000","last_active":"2021-08-21T15:00:21.740687+0000","last_peered":"2021-08-21T15:00:11.535600+0000","last_clean":"2021-08-21T15:00:11.535600+0000","last_became_active":"2021-08-21T15:00:10.517549+0000","last_became_peered":"2021-08-21T15:00:10.517549+0000","last_unstale":"2021-08-21T15:00:21.740687+0000","last_undegraded":"2021-08-21T15:00:21.740687+0000","last_fullsized":"2021-08-21T15:00:21.740687+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2021-08-21T15:00:11.891325+0000","last_change":"2021-08-21T15:00:10.516651+0000","last_active":"2021-08-21T15:00:11.891325+0000","last_peered":"2021-08-21T15:00:11.891325+0000","last_clean":"2021-08-21T15:00:11.891325+0000","last_became_active":"2021-08-21T15:00:10.516390+0000","last_became_peered":"2021-08-21T15:00:10.516390+0000","last_unstale":"2021-08-21T15:00:11.891325+0000","last_undegraded":"2021-08-21T15:00:11.891325+0000","last_fullsized":"2021-08-21T15:00:11.891325+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.740663+0000","last_change":"2021-08-21T15:00:21.741173+0000","last_active":"2021-08-21T15:00:21.740663+0000","last_peered":"2021-08-21T15:00:11.535607+0000","last_clean":"2021-08-21T15:00:11.535607+0000","last_became_active":"2021-08-21T15:00:10.517033+0000","last_became_peered":"2021-08-21T15:00:10.517033+0000","last_unstale":"2021-08-21T15:00:21.740663+0000","last_undegraded":"2021-08-21T15:00:21.740663+0000","last_fullsized":"2021-08-21T15:00:21.740663+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":3,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.734728+0000","last_change":"2021-08-21T15:00:21.735069+0000","last_active":"2021-08-21T15:00:09.483517+0000","last_peered":"2021-08-21T15:00:09.483517+0000","last_clean":"2021-08-21T15:00:09.483517+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2021-08-21T15:00:21.734728+0000","last_undegraded":"2021-08-21T15:00:21.734728+0000","last_fullsized":"2021-08-21T15:00:21.734728+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.740520+0000","last_change":"2021-08-21T15:00:21.740520+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2021-08-21T15:00:21.740520+0000","last_undegraded":"2021-08-21T15:00:21.740520+0000","last_fullsized":"2021-08-21T15:00:21.740520+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.742166+0000","last_change":"2021-08-21T15:00:21.742352+0000","last_active":"2021-08-21T15:00:21.742166+0000","last_peered":"2021-08-21T15:00:11.535480+0000","last_clean":"2021-08-21T15:00:11.535480+0000","last_became_active":"2021-08-21T15:00:10.516993+0000","last_became_peered":"2021-08-21T15:00:10.516993+0000","last_unstale":"2021-08-21T15:00:21.742166+0000","last_undegraded":"2021-08-21T15:00:21.742166+0000","last_fullsized":"2021-08-21T15:00:21.742166+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":1,"reported_epoch":14,"state":"peering","last_fresh":"2021-08-21T15:00:21.738352+0000","last_change":"2021-08-21T15:00:21.738352+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2021-08-21T15:00:21.738352+0000","last_undegraded":"2021-08-21T15:00:21.738352+0000","last_fullsized":"2021-08-21T15:00:21.738352+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672966,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4984,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366856,"statfs":{"total":96636764160,"available":96631660544,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":2,"up_from":10,"seq":42949672967,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4920,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366920,"statfs":{"total":96636764160,"available":96631726080,"internally_reserved":0,"allocated":114688,"data_stored":27420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]}]},{"osd":1,"up_from":0,"seq":42949672966,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705671,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:00:28.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:00:28.665 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:00:28.667 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:00:28.680 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2021-08-21T15:00:27.448028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":10000,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":9728,"kb_avail":188733680,"statfs":{"total":193273528320,"available":193263288320,"internally_reserved":0,"allocated":262144,"data_stored":65854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":9961472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2021-08-21T15:00:22.749343+0000","last_change":"2021-08-21T15:00:22.749343+0000","last_active":"2021-08-21T15:00:22.749343+0000","last_peered":"2021-08-21T15:00:22.749343+0000","last_clean":"2021-08-21T15:00:22.749343+0000","last_became_active":"2021-08-21T15:00:22.747990+0000","last_became_peered":"2021-08-21T15:00:22.747990+0000","last_unstale":"2021-08-21T15:00:22.749343+0000","last_undegraded":"2021-08-21T15:00:22.749343+0000","last_fullsized":"2021-08-21T15:00:22.749343+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2021-08-21T15:00:22.748465+0000","last_change":"2021-08-21T15:00:22.748465+0000","last_active":"2021-08-21T15:00:22.748465+0000","last_peered":"2021-08-21T15:00:22.748465+0000","last_clean":"2021-08-21T15:00:22.748465+0000","last_became_active":"2021-08-21T15:00:22.747936+0000","last_became_peered":"2021-08-21T15:00:22.747936+0000","last_unstale":"2021-08-21T15:00:22.748465+0000","last_undegraded":"2021-08-21T15:00:22.748465+0000","last_fullsized":"2021-08-21T15:00:22.748465+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-08-21T15:00:21.734301+0000","last_change":"2021-08-21T15:00:10.516651+0000","last_active":"2021-08-21T15:00:21.734301+0000","last_peered":"2021-08-21T15:00:21.734301+0000","last_clean":"2021-08-21T15:00:21.734301+0000","last_became_active":"2021-08-21T15:00:10.516390+0000","last_became_peered":"2021-08-21T15:00:10.516390+0000","last_unstale":"2021-08-21T15:00:21.734301+0000","last_undegraded":"2021-08-21T15:00:21.734301+0000","last_fullsized":"2021-08-21T15:00:21.734301+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2021-08-21T15:00:22.746299+0000","last_change":"2021-08-21T15:00:22.746299+0000","last_active":"2021-08-21T15:00:22.746299+0000","last_peered":"2021-08-21T15:00:22.746299+0000","last_clean":"2021-08-21T15:00:22.746299+0000","last_became_active":"2021-08-21T15:00:22.745984+0000","last_became_peered":"2021-08-21T15:00:22.745984+0000","last_unstale":"2021-08-21T15:00:22.746299+0000","last_undegraded":"2021-08-21T15:00:22.746299+0000","last_fullsized":"2021-08-21T15:00:22.746299+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2021-08-21T15:00:22.746999+0000","last_change":"2021-08-21T15:00:22.746999+0000","last_active":"2021-08-21T15:00:22.746999+0000","last_peered":"2021-08-21T15:00:22.746999+0000","last_clean":"2021-08-21T15:00:22.746999+0000","last_became_active":"2021-08-21T15:00:22.746693+0000","last_became_peered":"2021-08-21T15:00:22.746693+0000","last_unstale":"2021-08-21T15:00:22.746999+0000","last_undegraded":"2021-08-21T15:00:22.746999+0000","last_fullsized":"2021-08-21T15:00:22.746999+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2021-08-21T15:00:22.745792+0000","last_change":"2021-08-21T15:00:22.745792+0000","last_active":"2021-08-21T15:00:22.745792+0000","last_peered":"2021-08-21T15:00:22.745792+0000","last_clean":"2021-08-21T15:00:22.745792+0000","last_became_active":"2021-08-21T15:00:22.745452+0000","last_became_peered":"2021-08-21T15:00:22.745452+0000","last_unstale":"2021-08-21T15:00:22.745792+0000","last_undegraded":"2021-08-21T15:00:22.745792+0000","last_fullsized":"2021-08-21T15:00:22.745792+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2021-08-21T15:00:22.748958+0000","last_change":"2021-08-21T15:00:22.748958+0000","last_active":"2021-08-21T15:00:22.748958+0000","last_peered":"2021-08-21T15:00:22.748958+0000","last_clean":"2021-08-21T15:00:22.748958+0000","last_became_active":"2021-08-21T15:00:22.748106+0000","last_became_peered":"2021-08-21T15:00:22.748106+0000","last_unstale":"2021-08-21T15:00:22.748958+0000","last_undegraded":"2021-08-21T15:00:22.748958+0000","last_fullsized":"2021-08-21T15:00:22.748958+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2021-08-21T15:00:22.746463+0000","last_change":"2021-08-21T15:00:22.746463+0000","last_active":"2021-08-21T15:00:22.746463+0000","last_peered":"2021-08-21T15:00:22.746463+0000","last_clean":"2021-08-21T15:00:22.746463+0000","last_became_active":"2021-08-21T15:00:22.745861+0000","last_became_peered":"2021-08-21T15:00:22.745861+0000","last_unstale":"2021-08-21T15:00:22.746463+0000","last_undegraded":"2021-08-21T15:00:22.746463+0000","last_fullsized":"2021-08-21T15:00:22.746463+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672967,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5000,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366840,"statfs":{"total":96636764160,"available":96631644160,"internally_reserved":0,"allocated":131072,"data_stored":32927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":2,"up_from":10,"seq":42949672968,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5000,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":4864,"kb_avail":94366840,"statfs":{"total":96636764160,"available":96631644160,"internally_reserved":0,"allocated":131072,"data_stored":32927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4980736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]}]},{"osd":1,"up_from":0,"seq":42949672967,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705672,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:00:28.682 INFO:tasks.osd_recovery.ceph_manager:clean! 2021-08-21T15:00:28.682 INFO:tasks.osd_recovery:rados -p rbd bench 20 write -b 1 --no-cleanup 2021-08-21T15:00:28.682 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd bench 20 write -b 1 --no-cleanup 2021-08-21T15:00:28.724 INFO:teuthology.orchestra.run.smithi165.stdout:hints = 1 2021-08-21T15:00:28.725 INFO:teuthology.orchestra.run.smithi165.stdout:Maintaining 16 concurrent writes of 1 bytes to objects of size 1 for up to 20 seconds or 0 objects 2021-08-21T15:00:28.725 INFO:teuthology.orchestra.run.smithi165.stdout:Object prefix: benchmark_data_smithi165_21186 2021-08-21T15:00:28.725 INFO:teuthology.orchestra.run.smithi165.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-08-21T15:00:28.726 INFO:teuthology.orchestra.run.smithi165.stdout: 0 0 0 0 0 0 - 0 2021-08-21T15:00:29.724 INFO:teuthology.orchestra.run.smithi165.stdout: 1 16 2569 2553 0.0024346 0.00243473 0.00570959 0.00624225 2021-08-21T15:00:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: 2 16 4917 4901 0.00233649 0.00223923 0.00880128 0.00651546 2021-08-21T15:00:31.725 INFO:teuthology.orchestra.run.smithi165.stdout: 3 16 7104 7088 0.00225284 0.00208569 0.00865334 0.00676075 2021-08-21T15:00:32.725 INFO:teuthology.orchestra.run.smithi165.stdout: 4 16 9093 9077 0.00216381 0.00189686 0.00498191 0.00703777 2021-08-21T15:00:33.725 INFO:teuthology.orchestra.run.smithi165.stdout: 5 16 11058 11042 0.00210582 0.00187397 0.00791428 0.00723587 2021-08-21T15:00:34.725 INFO:teuthology.orchestra.run.smithi165.stdout: 6 16 12964 12948 0.00205778 0.0018177 0.012438 0.00740631 2021-08-21T15:00:35.725 INFO:teuthology.orchestra.run.smithi165.stdout: 7 16 14735 14719 0.00200507 0.00168896 0.0112174 0.00760248 2021-08-21T15:00:36.726 INFO:teuthology.orchestra.run.smithi165.stdout: 8 16 16417 16401 0.00195467 0.00160408 0.00966762 0.00779685 2021-08-21T15:00:37.727 INFO:teuthology.orchestra.run.smithi165.stdout: 9 16 17982 17966 0.00190321 0.0014925 0.0159366 0.00800402 2021-08-21T15:00:38.728 INFO:teuthology.orchestra.run.smithi165.stdout: 10 16 19478 19462 0.00185536 0.0014267 0.00784512 0.00821503 2021-08-21T15:00:39.728 INFO:teuthology.orchestra.run.smithi165.stdout: 11 16 20907 20891 0.00181058 0.0013628 0.0101105 0.00842012 2021-08-21T15:00:40.728 INFO:teuthology.orchestra.run.smithi165.stdout: 12 16 22280 22264 0.00176882 0.00130939 0.0126003 0.0086183 2021-08-21T15:00:41.728 INFO:teuthology.orchestra.run.smithi165.stdout: 13 16 23632 23616 0.00173194 0.00128937 0.018852 0.00880225 2021-08-21T15:00:42.728 INFO:teuthology.orchestra.run.smithi165.stdout: 14 16 24907 24891 0.00169508 0.00121593 0.00989379 0.00899361 2021-08-21T15:00:43.728 INFO:teuthology.orchestra.run.smithi165.stdout: 15 16 26129 26113 0.00165977 0.00116539 0.0105441 0.00918578 2021-08-21T15:00:44.728 INFO:teuthology.orchestra.run.smithi165.stdout: 16 16 27314 27298 0.00162666 0.0011301 0.00868643 0.00937258 2021-08-21T15:00:45.730 INFO:teuthology.orchestra.run.smithi165.stdout: 17 16 28467 28451 0.0015955 0.00109959 0.0111843 0.00955667 2021-08-21T15:00:46.730 INFO:teuthology.orchestra.run.smithi165.stdout: 18 16 29593 29577 0.00156652 0.00107384 0.0150871 0.00973416 2021-08-21T15:00:47.730 INFO:teuthology.orchestra.run.smithi165.stdout: 19 16 30671 30655 0.00153817 0.00102806 0.0149659 0.00990647 2021-08-21T15:00:48.730 INFO:teuthology.orchestra.run.smithi165.stdout:2021-08-21T15:00:48.733730+0000 min lat: 0.00255526 max lat: 0.112711 avg lat: 0.0101112 2021-08-21T15:00:48.731 INFO:teuthology.orchestra.run.smithi165.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-08-21T15:00:48.731 INFO:teuthology.orchestra.run.smithi165.stdout: 20 6 31645 31639 0.00150819 0.000938416 0.00741126 0.0101112 2021-08-21T15:00:49.731 INFO:teuthology.orchestra.run.smithi165.stdout:Total time run: 20.0143 2021-08-21T15:00:49.731 INFO:teuthology.orchestra.run.smithi165.stdout:Total writes made: 31645 2021-08-21T15:00:49.732 INFO:teuthology.orchestra.run.smithi165.stdout:Write size: 1 2021-08-21T15:00:49.732 INFO:teuthology.orchestra.run.smithi165.stdout:Object size: 1 2021-08-21T15:00:49.732 INFO:teuthology.orchestra.run.smithi165.stdout:Bandwidth (MB/sec): 0.00150788 2021-08-21T15:00:49.732 INFO:teuthology.orchestra.run.smithi165.stdout:Stddev Bandwidth: 0.000430484 2021-08-21T15:00:49.732 INFO:teuthology.orchestra.run.smithi165.stdout:Max bandwidth (MB/sec): 0.00243473 2021-08-21T15:00:49.733 INFO:teuthology.orchestra.run.smithi165.stdout:Min bandwidth (MB/sec): 0.000938416 2021-08-21T15:00:49.733 INFO:teuthology.orchestra.run.smithi165.stdout:Average IOPS: 1581 2021-08-21T15:00:49.733 INFO:teuthology.orchestra.run.smithi165.stdout:Stddev IOPS: 451.395 2021-08-21T15:00:49.733 INFO:teuthology.orchestra.run.smithi165.stdout:Max IOPS: 2553 2021-08-21T15:00:49.733 INFO:teuthology.orchestra.run.smithi165.stdout:Min IOPS: 984 2021-08-21T15:00:49.734 INFO:teuthology.orchestra.run.smithi165.stdout:Average Latency(s): 0.0101124 2021-08-21T15:00:49.734 INFO:teuthology.orchestra.run.smithi165.stdout:Stddev Latency(s): 0.00462727 2021-08-21T15:00:49.734 INFO:teuthology.orchestra.run.smithi165.stdout:Max latency(s): 0.112711 2021-08-21T15:00:49.734 INFO:teuthology.orchestra.run.smithi165.stdout:Min latency(s): 0.00255526 2021-08-21T15:00:49.742 INFO:tasks.osd_recovery:rados -p rbd put foo.1 /etc/passwd 2021-08-21T15:00:49.742 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.1 /etc/passwd 2021-08-21T15:00:49.840 INFO:tasks.osd_recovery:rados -p rbd put foo.2 /etc/passwd 2021-08-21T15:00:49.841 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.2 /etc/passwd 2021-08-21T15:00:49.893 INFO:tasks.osd_recovery:rados -p rbd put foo.3 /etc/passwd 2021-08-21T15:00:49.893 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.3 /etc/passwd 2021-08-21T15:00:49.944 INFO:tasks.osd_recovery:rados -p rbd put foo.4 /etc/passwd 2021-08-21T15:00:49.944 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.4 /etc/passwd 2021-08-21T15:00:49.997 INFO:tasks.osd_recovery:rados -p rbd put foo.5 /etc/passwd 2021-08-21T15:00:49.997 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.5 /etc/passwd 2021-08-21T15:00:50.050 INFO:tasks.osd_recovery:rados -p rbd put foo.6 /etc/passwd 2021-08-21T15:00:50.093 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.6 /etc/passwd 2021-08-21T15:00:50.160 INFO:tasks.osd_recovery:rados -p rbd put foo.7 /etc/passwd 2021-08-21T15:00:50.161 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.7 /etc/passwd 2021-08-21T15:00:50.217 INFO:tasks.osd_recovery:rados -p rbd put foo.8 /etc/passwd 2021-08-21T15:00:50.218 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.8 /etc/passwd 2021-08-21T15:00:50.277 INFO:tasks.osd_recovery:rados -p rbd put foo.9 /etc/passwd 2021-08-21T15:00:50.278 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.9 /etc/passwd 2021-08-21T15:00:50.373 INFO:tasks.osd_recovery:rados -p rbd put foo.10 /etc/passwd 2021-08-21T15:00:50.373 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.10 /etc/passwd 2021-08-21T15:00:50.424 INFO:tasks.osd_recovery:rados -p rbd put foo.11 /etc/passwd 2021-08-21T15:00:50.425 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.11 /etc/passwd 2021-08-21T15:00:50.477 INFO:tasks.osd_recovery:rados -p rbd put foo.12 /etc/passwd 2021-08-21T15:00:50.477 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.12 /etc/passwd 2021-08-21T15:00:50.529 INFO:tasks.osd_recovery:rados -p rbd put foo.13 /etc/passwd 2021-08-21T15:00:50.529 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.13 /etc/passwd 2021-08-21T15:00:50.592 INFO:tasks.osd_recovery:rados -p rbd put foo.14 /etc/passwd 2021-08-21T15:00:50.592 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.14 /etc/passwd 2021-08-21T15:00:50.644 INFO:tasks.osd_recovery:rados -p rbd put foo.15 /etc/passwd 2021-08-21T15:00:50.644 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.15 /etc/passwd 2021-08-21T15:00:50.698 INFO:tasks.osd_recovery:rados -p rbd put foo.16 /etc/passwd 2021-08-21T15:00:50.698 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.16 /etc/passwd 2021-08-21T15:00:50.749 INFO:tasks.osd_recovery:rados -p rbd put foo.17 /etc/passwd 2021-08-21T15:00:50.750 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.17 /etc/passwd 2021-08-21T15:00:50.801 INFO:tasks.osd_recovery:rados -p rbd put foo.18 /etc/passwd 2021-08-21T15:00:50.801 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.18 /etc/passwd 2021-08-21T15:00:50.857 INFO:tasks.osd_recovery:rados -p rbd put foo.19 /etc/passwd 2021-08-21T15:00:50.857 DEBUG:teuthology.orchestra.run.smithi165:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.19 /etc/passwd 2021-08-21T15:00:50.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 1 2021-08-21T15:00:51.183 INFO:tasks.ceph.mon.a.smithi165.stderr:2021-08-21T15:00:51.181+0000 7f927d985700 -1 mon.a@0(leader).osd e15 definitely_dead 0 2021-08-21T15:00:51.741 INFO:teuthology.orchestra.run.smithi165.stderr:marked in osd.0. marked in osd.1. 2021-08-21T15:00:51.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 3 2021-08-21T15:00:52.031 INFO:tasks.ceph.mon.a.smithi165.stderr:2021-08-21T15:00:52.029+0000 7f927d985700 -1 mon.a@0(leader).osd e16 definitely_dead 0 2021-08-21T15:00:52.845 INFO:teuthology.orchestra.run.smithi165.stderr:marked out osd.2. marked out osd.3. 2021-08-21T15:01:02.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-08-21T15:01:03.013 INFO:teuthology.orchestra.run.smithi165.stdout:38654705680 2021-08-21T15:01:03.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-08-21T15:01:03.161 INFO:teuthology.orchestra.run.smithi165.stdout:42949672975 2021-08-21T15:01:03.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-08-21T15:01:03.310 INFO:teuthology.orchestra.run.smithi165.stdout:42949672976 2021-08-21T15:01:03.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2021-08-21T15:01:03.453 INFO:teuthology.orchestra.run.smithi165.stdout:42949672975 2021-08-21T15:01:03.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-08-21T15:01:03.729 INFO:teuthology.orchestra.run.smithi165.stdout:38654705679 2021-08-21T15:01:03.749 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705680 got 38654705679 for osd.0 2021-08-21T15:01:04.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-08-21T15:01:05.034 INFO:teuthology.orchestra.run.smithi165.stdout:38654705680 2021-08-21T15:01:05.052 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705680 got 38654705680 for osd.0 2021-08-21T15:01:05.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-08-21T15:01:05.321 INFO:teuthology.orchestra.run.smithi165.stdout:42949672975 2021-08-21T15:01:05.340 INFO:tasks.osd_recovery.ceph_manager:need seq 42949672975 got 42949672975 for osd.1 2021-08-21T15:01:15.342 INFO:tasks.osd_recovery.ceph_manager:waiting for peering to complete 2021-08-21T15:01:15.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:01:15.617 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:15.619 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:01:15.635 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2021-08-21T15:01:15.460323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27679,"num_objects_misplaced":35651,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":40144,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":188703536,"statfs":{"total":193273528320,"available":193232420864,"internally_reserved":0,"allocated":327680,"data_stored":89294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004130"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3975,"reported_epoch":19,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.996880+0000","last_change":"2021-08-21T15:00:54.997848+0000","last_active":"2021-08-21T15:00:54.996880+0000","last_peered":"2021-08-21T15:00:54.996880+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:00:54.993830+0000","last_became_peered":"2021-08-21T15:00:54.993830+0000","last_unstale":"2021-08-21T15:00:54.996880+0000","last_undegraded":"2021-08-21T15:00:54.807475+0000","last_fullsized":"2021-08-21T15:00:54.712047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3927,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.352667+0000","last_change":"2021-08-21T15:00:54.352667+0000","last_active":"2021-08-21T15:00:54.352667+0000","last_peered":"2021-08-21T15:00:54.352667+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:00:54.343677+0000","last_became_peered":"2021-08-21T15:00:54.343677+0000","last_unstale":"2021-08-21T15:00:54.352667+0000","last_undegraded":"2021-08-21T15:00:53.942467+0000","last_fullsized":"2021-08-21T15:00:53.714751+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3880,"reported_epoch":19,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:55.018369+0000","last_change":"2021-08-21T15:00:55.019285+0000","last_active":"2021-08-21T15:00:55.018369+0000","last_peered":"2021-08-21T15:00:55.018369+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:00:55.014941+0000","last_became_peered":"2021-08-21T15:00:55.014941+0000","last_unstale":"2021-08-21T15:00:55.018369+0000","last_undegraded":"2021-08-21T15:00:54.855081+0000","last_fullsized":"2021-08-21T15:00:54.711968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4018,"reported_epoch":18,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:00:54.268728+0000","last_change":"2021-08-21T15:00:54.721037+0000","last_active":"2021-08-21T15:00:54.268728+0000","last_peered":"2021-08-21T15:00:54.268728+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:00:54.261113+0000","last_became_peered":"2021-08-21T15:00:54.261113+0000","last_unstale":"2021-08-21T15:00:54.268728+0000","last_undegraded":"2021-08-21T15:00:54.017884+0000","last_fullsized":"2021-08-21T15:00:53.714875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7972,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4009,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.349527+0000","last_change":"2021-08-21T15:00:54.349527+0000","last_active":"2021-08-21T15:00:54.349527+0000","last_peered":"2021-08-21T15:00:54.349527+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:00:54.341544+0000","last_became_peered":"2021-08-21T15:00:54.341544+0000","last_unstale":"2021-08-21T15:00:54.349527+0000","last_undegraded":"2021-08-21T15:00:54.007259+0000","last_fullsized":"2021-08-21T15:00:53.709556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":3941,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.348238+0000","last_change":"2021-08-21T15:00:54.348238+0000","last_active":"2021-08-21T15:00:54.348238+0000","last_peered":"2021-08-21T15:00:54.348238+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:00:54.340749+0000","last_became_peered":"2021-08-21T15:00:54.340749+0000","last_unstale":"2021-08-21T15:00:54.348238+0000","last_undegraded":"2021-08-21T15:00:53.916952+0000","last_fullsized":"2021-08-21T15:00:53.709537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3923,"num_objects_misplaced":3923,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3923}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4067,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.351206+0000","last_change":"2021-08-21T15:00:54.351206+0000","last_active":"2021-08-21T15:00:54.351206+0000","last_peered":"2021-08-21T15:00:54.351206+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:00:54.342124+0000","last_became_peered":"2021-08-21T15:00:54.342124+0000","last_unstale":"2021-08-21T15:00:54.351206+0000","last_undegraded":"2021-08-21T15:00:54.090098+0000","last_fullsized":"2021-08-21T15:00:53.714751+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4040,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.318004+0000","last_change":"2021-08-21T15:00:54.318004+0000","last_active":"2021-08-21T15:00:54.318004+0000","last_peered":"2021-08-21T15:00:54.318004+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:00:54.313821+0000","last_became_peered":"2021-08-21T15:00:54.313821+0000","last_unstale":"2021-08-21T15:00:54.318004+0000","last_undegraded":"2021-08-21T15:00:53.989853+0000","last_fullsized":"2021-08-21T15:00:53.709615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27679,"num_objects_misplaced":35651,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":259555328,"data_stored":411000,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":42949672978,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":42949672979,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672978,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20072,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":94351768,"statfs":{"total":96636764160,"available":96616210432,"internally_reserved":0,"allocated":163840,"data_stored":44647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.78400000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.70499999999999996}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.63},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.751}]}]},{"osd":0,"up_from":9,"seq":38654705683,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20072,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":94351768,"statfs":{"total":96636764160,"available":96616210432,"internally_reserved":0,"allocated":163840,"data_stored":44647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:01:15.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:01:15.899 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:15.901 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:01:15.911 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2021-08-21T15:01:15.460323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27679,"num_objects_misplaced":35651,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":40144,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":188703536,"statfs":{"total":193273528320,"available":193232420864,"internally_reserved":0,"allocated":327680,"data_stored":89294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004130"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3975,"reported_epoch":19,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.996880+0000","last_change":"2021-08-21T15:00:54.997848+0000","last_active":"2021-08-21T15:00:54.996880+0000","last_peered":"2021-08-21T15:00:54.996880+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:00:54.993830+0000","last_became_peered":"2021-08-21T15:00:54.993830+0000","last_unstale":"2021-08-21T15:00:54.996880+0000","last_undegraded":"2021-08-21T15:00:54.807475+0000","last_fullsized":"2021-08-21T15:00:54.712047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3927,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.352667+0000","last_change":"2021-08-21T15:00:54.352667+0000","last_active":"2021-08-21T15:00:54.352667+0000","last_peered":"2021-08-21T15:00:54.352667+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:00:54.343677+0000","last_became_peered":"2021-08-21T15:00:54.343677+0000","last_unstale":"2021-08-21T15:00:54.352667+0000","last_undegraded":"2021-08-21T15:00:53.942467+0000","last_fullsized":"2021-08-21T15:00:53.714751+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3880,"reported_epoch":19,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:55.018369+0000","last_change":"2021-08-21T15:00:55.019285+0000","last_active":"2021-08-21T15:00:55.018369+0000","last_peered":"2021-08-21T15:00:55.018369+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:00:55.014941+0000","last_became_peered":"2021-08-21T15:00:55.014941+0000","last_unstale":"2021-08-21T15:00:55.018369+0000","last_undegraded":"2021-08-21T15:00:54.855081+0000","last_fullsized":"2021-08-21T15:00:54.711968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4018,"reported_epoch":18,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:00:54.268728+0000","last_change":"2021-08-21T15:00:54.721037+0000","last_active":"2021-08-21T15:00:54.268728+0000","last_peered":"2021-08-21T15:00:54.268728+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:00:54.261113+0000","last_became_peered":"2021-08-21T15:00:54.261113+0000","last_unstale":"2021-08-21T15:00:54.268728+0000","last_undegraded":"2021-08-21T15:00:54.017884+0000","last_fullsized":"2021-08-21T15:00:53.714875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7972,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4009,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.349527+0000","last_change":"2021-08-21T15:00:54.349527+0000","last_active":"2021-08-21T15:00:54.349527+0000","last_peered":"2021-08-21T15:00:54.349527+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:00:54.341544+0000","last_became_peered":"2021-08-21T15:00:54.341544+0000","last_unstale":"2021-08-21T15:00:54.349527+0000","last_undegraded":"2021-08-21T15:00:54.007259+0000","last_fullsized":"2021-08-21T15:00:53.709556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":3941,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.348238+0000","last_change":"2021-08-21T15:00:54.348238+0000","last_active":"2021-08-21T15:00:54.348238+0000","last_peered":"2021-08-21T15:00:54.348238+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:00:54.340749+0000","last_became_peered":"2021-08-21T15:00:54.340749+0000","last_unstale":"2021-08-21T15:00:54.348238+0000","last_undegraded":"2021-08-21T15:00:53.916952+0000","last_fullsized":"2021-08-21T15:00:53.709537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3923,"num_objects_misplaced":3923,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3923}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4067,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.351206+0000","last_change":"2021-08-21T15:00:54.351206+0000","last_active":"2021-08-21T15:00:54.351206+0000","last_peered":"2021-08-21T15:00:54.351206+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:00:54.342124+0000","last_became_peered":"2021-08-21T15:00:54.342124+0000","last_unstale":"2021-08-21T15:00:54.351206+0000","last_undegraded":"2021-08-21T15:00:54.090098+0000","last_fullsized":"2021-08-21T15:00:53.714751+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4040,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.318004+0000","last_change":"2021-08-21T15:00:54.318004+0000","last_active":"2021-08-21T15:00:54.318004+0000","last_peered":"2021-08-21T15:00:54.318004+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:00:54.313821+0000","last_became_peered":"2021-08-21T15:00:54.313821+0000","last_unstale":"2021-08-21T15:00:54.318004+0000","last_undegraded":"2021-08-21T15:00:53.989853+0000","last_fullsized":"2021-08-21T15:00:53.709615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27679,"num_objects_misplaced":35651,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":259555328,"data_stored":411000,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":42949672978,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":42949672979,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672978,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20072,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":94351768,"statfs":{"total":96636764160,"available":96616210432,"internally_reserved":0,"allocated":163840,"data_stored":44647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.78400000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.70499999999999996}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.63},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.751}]}]},{"osd":0,"up_from":9,"seq":38654705683,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20072,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":94351768,"statfs":{"total":96636764160,"available":96616210432,"internally_reserved":0,"allocated":163840,"data_stored":44647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:01:15.913 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:01:16.232 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:16.232 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27679/63330 objects degraded (43.706%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":76,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558008,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":41107456,"bytes_avail":193232420864,"bytes_total":193273528320,"degraded_objects":27679,"degraded_total":63330,"degraded_ratio":0.4370598452550134,"misplaced_objects":35651,"misplaced_total":63330,"misplaced_ratio":0.5629401547449866},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2021-08-21T15:01:11.459247+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (20s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (20s)\n [............................] ","progress":0,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (20s)\n [............................] ","progress":0,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (15s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (20s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:01:16.245 INFO:tasks.osd_recovery.ceph_manager:{'fsid': 'b82e8492-200f-4d81-953f-d1ffd38e4baa', 'health': {'status': 'HEALTH_WARN', 'checks': {'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 27679/63330 objects degraded (43.706%), 7 pgs degraded', 'count': 7}, 'muted': False}}, 'mutes': []}, 'election_epoch': 12, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 76, 'monmap': {'epoch': 1, 'min_mon_release_name': 'quincy', 'num_mons': 3}, 'osdmap': {'epoch': 19, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1629558008, 'num_in_osds': 2, 'osd_in_since': 1629558052, 'num_remapped_pgs': 8}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_wait+undersized+degraded+remapped', 'count': 7}, {'state_name': 'active+recovering+undersized+remapped', 'count': 1}], 'num_pgs': 8, 'num_pools': 1, 'num_objects': 31665, 'data_bytes': 205500, 'bytes_used': 41107456, 'bytes_avail': 193232420864, 'bytes_total': 193273528320, 'degraded_objects': 27679, 'degraded_total': 63330, 'degraded_ratio': 0.4370598452550134, 'misplaced_objects': 35651, 'misplaced_total': 63330, 'misplaced_ratio': 0.5629401547449866}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 4, 'modified': '2021-08-21T15:01:11.459247+0000', 'services': {}}, 'progress_events': {'1d845a39-f91f-4700-a0e8-2c1b75ec5d3c': {'message': 'Rebalancing after osd.0 marked in (20s)\n [............................] ', 'progress': 0, 'add_to_ceph_s': False}, '5ec8e6e9-ddf9-4800-be89-db0fd109615b': {'message': 'Rebalancing after osd.2 marked out (20s)\n [............................] ', 'progress': 0, 'add_to_ceph_s': False}, 'b76d601d-b890-4d7a-9557-b6361aca0b52': {'message': 'Rebalancing after osd.3 marked out (20s)\n [............................] ', 'progress': 0, 'add_to_ceph_s': False}, 'e6fb09c2-bb5d-4094-b0ef-938c511d27c2': {'message': 'Global Recovery Event (15s)\n [............................] ', 'progress': 0, 'add_to_ceph_s': True}, 'ff8c034e-9d62-4aeb-bb21-acf47dc5d977': {'message': 'Rebalancing after osd.1 marked in (20s)\n [............................] ', 'progress': 0, 'add_to_ceph_s': False}}} 2021-08-21T15:01:16.246 INFO:tasks.osd_recovery.ceph_manager:active! 2021-08-21T15:01:16.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:01:16.509 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:16.511 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:01:16.521 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2021-08-21T15:01:15.460323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27679,"num_objects_misplaced":35651,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":40144,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":188703536,"statfs":{"total":193273528320,"available":193232420864,"internally_reserved":0,"allocated":327680,"data_stored":89294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004130"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3975,"reported_epoch":19,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.996880+0000","last_change":"2021-08-21T15:00:54.997848+0000","last_active":"2021-08-21T15:00:54.996880+0000","last_peered":"2021-08-21T15:00:54.996880+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:00:54.993830+0000","last_became_peered":"2021-08-21T15:00:54.993830+0000","last_unstale":"2021-08-21T15:00:54.996880+0000","last_undegraded":"2021-08-21T15:00:54.807475+0000","last_fullsized":"2021-08-21T15:00:54.712047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3927,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.352667+0000","last_change":"2021-08-21T15:00:54.352667+0000","last_active":"2021-08-21T15:00:54.352667+0000","last_peered":"2021-08-21T15:00:54.352667+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:00:54.343677+0000","last_became_peered":"2021-08-21T15:00:54.343677+0000","last_unstale":"2021-08-21T15:00:54.352667+0000","last_undegraded":"2021-08-21T15:00:53.942467+0000","last_fullsized":"2021-08-21T15:00:53.714751+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3880,"reported_epoch":19,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:55.018369+0000","last_change":"2021-08-21T15:00:55.019285+0000","last_active":"2021-08-21T15:00:55.018369+0000","last_peered":"2021-08-21T15:00:55.018369+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:00:55.014941+0000","last_became_peered":"2021-08-21T15:00:55.014941+0000","last_unstale":"2021-08-21T15:00:55.018369+0000","last_undegraded":"2021-08-21T15:00:54.855081+0000","last_fullsized":"2021-08-21T15:00:54.711968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4018,"reported_epoch":18,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:00:54.268728+0000","last_change":"2021-08-21T15:00:54.721037+0000","last_active":"2021-08-21T15:00:54.268728+0000","last_peered":"2021-08-21T15:00:54.268728+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:00:54.261113+0000","last_became_peered":"2021-08-21T15:00:54.261113+0000","last_unstale":"2021-08-21T15:00:54.268728+0000","last_undegraded":"2021-08-21T15:00:54.017884+0000","last_fullsized":"2021-08-21T15:00:53.714875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7972,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4009,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.349527+0000","last_change":"2021-08-21T15:00:54.349527+0000","last_active":"2021-08-21T15:00:54.349527+0000","last_peered":"2021-08-21T15:00:54.349527+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:00:54.341544+0000","last_became_peered":"2021-08-21T15:00:54.341544+0000","last_unstale":"2021-08-21T15:00:54.349527+0000","last_undegraded":"2021-08-21T15:00:54.007259+0000","last_fullsized":"2021-08-21T15:00:53.709556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":3941,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.348238+0000","last_change":"2021-08-21T15:00:54.348238+0000","last_active":"2021-08-21T15:00:54.348238+0000","last_peered":"2021-08-21T15:00:54.348238+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:00:54.340749+0000","last_became_peered":"2021-08-21T15:00:54.340749+0000","last_unstale":"2021-08-21T15:00:54.348238+0000","last_undegraded":"2021-08-21T15:00:53.916952+0000","last_fullsized":"2021-08-21T15:00:53.709537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3923,"num_objects_misplaced":3923,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3923}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4067,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.351206+0000","last_change":"2021-08-21T15:00:54.351206+0000","last_active":"2021-08-21T15:00:54.351206+0000","last_peered":"2021-08-21T15:00:54.351206+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:00:54.342124+0000","last_became_peered":"2021-08-21T15:00:54.342124+0000","last_unstale":"2021-08-21T15:00:54.351206+0000","last_undegraded":"2021-08-21T15:00:54.090098+0000","last_fullsized":"2021-08-21T15:00:53.714751+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4040,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.318004+0000","last_change":"2021-08-21T15:00:54.318004+0000","last_active":"2021-08-21T15:00:54.318004+0000","last_peered":"2021-08-21T15:00:54.318004+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:00:54.313821+0000","last_became_peered":"2021-08-21T15:00:54.313821+0000","last_unstale":"2021-08-21T15:00:54.318004+0000","last_undegraded":"2021-08-21T15:00:53.989853+0000","last_fullsized":"2021-08-21T15:00:53.709615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27679,"num_objects_misplaced":35651,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":259555328,"data_stored":411000,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":42949672978,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":42949672979,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672978,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20072,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":94351768,"statfs":{"total":96636764160,"available":96616210432,"internally_reserved":0,"allocated":163840,"data_stored":44647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.78400000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.70499999999999996}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.63},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.751}]}]},{"osd":0,"up_from":9,"seq":38654705683,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20072,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":94351768,"statfs":{"total":96636764160,"available":96616210432,"internally_reserved":0,"allocated":163840,"data_stored":44647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:01:16.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:01:16.792 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:16.793 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:01:16.804 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2021-08-21T15:01:15.460323+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27679,"num_objects_misplaced":35651,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":40144,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":188703536,"statfs":{"total":193273528320,"available":193232420864,"internally_reserved":0,"allocated":327680,"data_stored":89294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004130"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3975,"reported_epoch":19,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.996880+0000","last_change":"2021-08-21T15:00:54.997848+0000","last_active":"2021-08-21T15:00:54.996880+0000","last_peered":"2021-08-21T15:00:54.996880+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:00:54.993830+0000","last_became_peered":"2021-08-21T15:00:54.993830+0000","last_unstale":"2021-08-21T15:00:54.996880+0000","last_undegraded":"2021-08-21T15:00:54.807475+0000","last_fullsized":"2021-08-21T15:00:54.712047+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3927,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.352667+0000","last_change":"2021-08-21T15:00:54.352667+0000","last_active":"2021-08-21T15:00:54.352667+0000","last_peered":"2021-08-21T15:00:54.352667+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:00:54.343677+0000","last_became_peered":"2021-08-21T15:00:54.343677+0000","last_unstale":"2021-08-21T15:00:54.352667+0000","last_undegraded":"2021-08-21T15:00:53.942467+0000","last_fullsized":"2021-08-21T15:00:53.714751+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3880,"reported_epoch":19,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:55.018369+0000","last_change":"2021-08-21T15:00:55.019285+0000","last_active":"2021-08-21T15:00:55.018369+0000","last_peered":"2021-08-21T15:00:55.018369+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:00:55.014941+0000","last_became_peered":"2021-08-21T15:00:55.014941+0000","last_unstale":"2021-08-21T15:00:55.018369+0000","last_undegraded":"2021-08-21T15:00:54.855081+0000","last_fullsized":"2021-08-21T15:00:54.711968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4018,"reported_epoch":18,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:00:54.268728+0000","last_change":"2021-08-21T15:00:54.721037+0000","last_active":"2021-08-21T15:00:54.268728+0000","last_peered":"2021-08-21T15:00:54.268728+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:00:54.261113+0000","last_became_peered":"2021-08-21T15:00:54.261113+0000","last_unstale":"2021-08-21T15:00:54.268728+0000","last_undegraded":"2021-08-21T15:00:54.017884+0000","last_fullsized":"2021-08-21T15:00:53.714875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7972,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4009,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.349527+0000","last_change":"2021-08-21T15:00:54.349527+0000","last_active":"2021-08-21T15:00:54.349527+0000","last_peered":"2021-08-21T15:00:54.349527+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:00:54.341544+0000","last_became_peered":"2021-08-21T15:00:54.341544+0000","last_unstale":"2021-08-21T15:00:54.349527+0000","last_undegraded":"2021-08-21T15:00:54.007259+0000","last_fullsized":"2021-08-21T15:00:53.709556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":3941,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.348238+0000","last_change":"2021-08-21T15:00:54.348238+0000","last_active":"2021-08-21T15:00:54.348238+0000","last_peered":"2021-08-21T15:00:54.348238+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:00:54.340749+0000","last_became_peered":"2021-08-21T15:00:54.340749+0000","last_unstale":"2021-08-21T15:00:54.348238+0000","last_undegraded":"2021-08-21T15:00:53.916952+0000","last_fullsized":"2021-08-21T15:00:53.709537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3923,"num_objects_misplaced":3923,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3923}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4067,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.351206+0000","last_change":"2021-08-21T15:00:54.351206+0000","last_active":"2021-08-21T15:00:54.351206+0000","last_peered":"2021-08-21T15:00:54.351206+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:00:54.342124+0000","last_became_peered":"2021-08-21T15:00:54.342124+0000","last_unstale":"2021-08-21T15:00:54.351206+0000","last_undegraded":"2021-08-21T15:00:54.090098+0000","last_fullsized":"2021-08-21T15:00:53.714751+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4040,"reported_epoch":18,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:00:54.318004+0000","last_change":"2021-08-21T15:00:54.318004+0000","last_active":"2021-08-21T15:00:54.318004+0000","last_peered":"2021-08-21T15:00:54.318004+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:00:54.313821+0000","last_became_peered":"2021-08-21T15:00:54.313821+0000","last_unstale":"2021-08-21T15:00:54.318004+0000","last_undegraded":"2021-08-21T15:00:53.989853+0000","last_fullsized":"2021-08-21T15:00:53.709615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27679,"num_objects_misplaced":35651,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":259555328,"data_stored":411000,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":42949672978,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":42949672979,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672978,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20072,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":94351768,"statfs":{"total":96636764160,"available":96616210432,"internally_reserved":0,"allocated":163840,"data_stored":44647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.78400000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.70499999999999996}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.63},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.751}]}]},{"osd":0,"up_from":9,"seq":38654705683,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20072,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":94351768,"statfs":{"total":96636764160,"available":96616210432,"internally_reserved":0,"allocated":163840,"data_stored":44647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:01:16.806 INFO:tasks.osd_recovery:stopping 2,3 2021-08-21T15:01:16.807 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2021-08-21T15:01:16.807 INFO:teuthology.orchestra.run:waiting for 300 2021-08-21T15:01:16.897 INFO:tasks.ceph.osd.2:Stopped 2021-08-21T15:01:16.898 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2021-08-21T15:01:16.899 INFO:teuthology.orchestra.run:waiting for 300 2021-08-21T15:01:16.976 INFO:tasks.ceph.osd.3:Stopped 2021-08-21T15:01:16.977 INFO:tasks.osd_recovery:... 2021-08-21T15:01:16.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 2 3 2021-08-21T15:01:17.270 INFO:tasks.ceph.mon.a.smithi165.stderr:2021-08-21T15:01:17.267+0000 7f927d985700 -1 mon.a@0(leader).osd e19 definitely_dead 0 2021-08-21T15:01:18.034 INFO:teuthology.orchestra.run.smithi165.stderr:marked down osd.2. marked down osd.3. 2021-08-21T15:01:18.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-08-21T15:01:18.196 INFO:teuthology.orchestra.run.smithi165.stdout:38654705684 2021-08-21T15:01:18.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-08-21T15:01:18.354 INFO:teuthology.orchestra.run.smithi165.stdout:42949672979 2021-08-21T15:01:18.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-08-21T15:01:18.627 INFO:teuthology.orchestra.run.smithi165.stdout:38654705683 2021-08-21T15:01:18.641 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705684 got 38654705683 for osd.0 2021-08-21T15:01:19.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-08-21T15:01:19.915 INFO:teuthology.orchestra.run.smithi165.stdout:38654705683 2021-08-21T15:01:19.932 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705684 got 38654705683 for osd.0 2021-08-21T15:01:20.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-08-21T15:01:21.214 INFO:teuthology.orchestra.run.smithi165.stdout:38654705685 2021-08-21T15:01:21.231 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705684 got 38654705685 for osd.0 2021-08-21T15:01:21.234 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-08-21T15:01:21.498 INFO:teuthology.orchestra.run.smithi165.stdout:42949672979 2021-08-21T15:01:21.512 INFO:tasks.osd_recovery.ceph_manager:need seq 42949672979 got 42949672979 for osd.1 2021-08-21T15:01:21.512 INFO:tasks.osd_recovery.ceph_manager:waiting for peering to complete or become blocked 2021-08-21T15:01:21.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:01:21.782 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:21.786 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:01:21.796 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":55,"stamp":"2021-08-21T15:01:21.462049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":31665,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":40160,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":188703520,"statfs":{"total":193273528320,"available":193232404480,"internally_reserved":0,"allocated":344064,"data_stored":94890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":31665,"num_objects_degraded":-27679,"num_objects_misplaced":-35651,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002248"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3973,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.029695+0000","last_change":"2021-08-21T15:01:18.029695+0000","last_active":"2021-08-21T15:00:52.843925+0000","last_peered":"2021-08-21T15:00:48.699483+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:00:22.747990+0000","last_became_peered":"2021-08-21T15:00:22.747990+0000","last_unstale":"2021-08-21T15:01:18.029695+0000","last_undegraded":"2021-08-21T15:01:18.029695+0000","last_fullsized":"2021-08-21T15:01:18.029695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":3956,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3923,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.028113+0000","last_change":"2021-08-21T15:01:18.028139+0000","last_active":"2021-08-21T15:00:52.409750+0000","last_peered":"2021-08-21T15:00:50.527252+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:00:22.747936+0000","last_became_peered":"2021-08-21T15:00:22.747936+0000","last_unstale":"2021-08-21T15:01:18.028113+0000","last_undegraded":"2021-08-21T15:01:18.028113+0000","last_fullsized":"2021-08-21T15:01:18.028113+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":3896,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3877,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.026287+0000","last_change":"2021-08-21T15:01:18.026329+0000","last_active":"2021-08-21T15:00:53.709677+0000","last_peered":"2021-08-21T15:00:52.399475+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:00:10.516390+0000","last_became_peered":"2021-08-21T15:00:10.516390+0000","last_unstale":"2021-08-21T15:01:18.026287+0000","last_undegraded":"2021-08-21T15:01:18.026287+0000","last_fullsized":"2021-08-21T15:01:18.026287+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":3859,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4013,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.027645+0000","last_change":"2021-08-21T15:01:18.027691+0000","last_active":"2021-08-21T15:00:52.413530+0000","last_peered":"2021-08-21T15:00:50.800815+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:00:22.745984+0000","last_became_peered":"2021-08-21T15:00:22.745984+0000","last_unstale":"2021-08-21T15:01:18.027645+0000","last_undegraded":"2021-08-21T15:01:18.027645+0000","last_fullsized":"2021-08-21T15:01:18.027645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":3986,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4006,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.030921+0000","last_change":"2021-08-21T15:01:18.030921+0000","last_active":"2021-08-21T15:00:52.843710+0000","last_peered":"2021-08-21T15:00:50.372465+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:00:22.746693+0000","last_became_peered":"2021-08-21T15:00:22.746693+0000","last_unstale":"2021-08-21T15:01:18.030921+0000","last_undegraded":"2021-08-21T15:01:18.030921+0000","last_fullsized":"2021-08-21T15:01:18.030921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":3988,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":3937,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.031224+0000","last_change":"2021-08-21T15:01:18.031261+0000","last_active":"2021-08-21T15:00:52.843632+0000","last_peered":"2021-08-21T15:00:50.696737+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:00:22.745452+0000","last_became_peered":"2021-08-21T15:00:22.745452+0000","last_unstale":"2021-08-21T15:01:18.031224+0000","last_undegraded":"2021-08-21T15:01:18.031224+0000","last_fullsized":"2021-08-21T15:01:18.031224+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":3923,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4063,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.026564+0000","last_change":"2021-08-21T15:01:18.026631+0000","last_active":"2021-08-21T15:00:52.409587+0000","last_peered":"2021-08-21T15:00:50.216409+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:00:22.748106+0000","last_became_peered":"2021-08-21T15:00:22.748106+0000","last_unstale":"2021-08-21T15:01:18.026564+0000","last_undegraded":"2021-08-21T15:01:18.026564+0000","last_fullsized":"2021-08-21T15:01:18.026564+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":4036,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4037,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.026735+0000","last_change":"2021-08-21T15:01:18.026735+0000","last_active":"2021-08-21T15:00:52.841845+0000","last_peered":"2021-08-21T15:00:50.856667+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:00:22.745861+0000","last_became_peered":"2021-08-21T15:00:22.745861+0000","last_unstale":"2021-08-21T15:01:18.026735+0000","last_undegraded":"2021-08-21T15:01:18.026735+0000","last_fullsized":"2021-08-21T15:01:18.026735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":4021,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":31665,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":259555328,"data_stored":411000,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":42949672978,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":42949672979,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672980,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20080,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":94351760,"statfs":{"total":96636764160,"available":96616202240,"internally_reserved":0,"allocated":172032,"data_stored":47445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.437}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.67000000000000004}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.61699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705685,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20080,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":94351760,"statfs":{"total":96636764160,"available":96616202240,"internally_reserved":0,"allocated":172032,"data_stored":47445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.35499999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:01:21.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:01:22.062 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:22.064 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:01:22.074 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":55,"stamp":"2021-08-21T15:01:21.462049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":31665,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":40160,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":188703520,"statfs":{"total":193273528320,"available":193232404480,"internally_reserved":0,"allocated":344064,"data_stored":94890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":31665,"num_objects_degraded":-27679,"num_objects_misplaced":-35651,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002248"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3973,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.029695+0000","last_change":"2021-08-21T15:01:18.029695+0000","last_active":"2021-08-21T15:00:52.843925+0000","last_peered":"2021-08-21T15:00:48.699483+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:00:22.747990+0000","last_became_peered":"2021-08-21T15:00:22.747990+0000","last_unstale":"2021-08-21T15:01:18.029695+0000","last_undegraded":"2021-08-21T15:01:18.029695+0000","last_fullsized":"2021-08-21T15:01:18.029695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":3956,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3923,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.028113+0000","last_change":"2021-08-21T15:01:18.028139+0000","last_active":"2021-08-21T15:00:52.409750+0000","last_peered":"2021-08-21T15:00:50.527252+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:00:22.747936+0000","last_became_peered":"2021-08-21T15:00:22.747936+0000","last_unstale":"2021-08-21T15:01:18.028113+0000","last_undegraded":"2021-08-21T15:01:18.028113+0000","last_fullsized":"2021-08-21T15:01:18.028113+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":3896,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3877,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.026287+0000","last_change":"2021-08-21T15:01:18.026329+0000","last_active":"2021-08-21T15:00:53.709677+0000","last_peered":"2021-08-21T15:00:52.399475+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:00:10.516390+0000","last_became_peered":"2021-08-21T15:00:10.516390+0000","last_unstale":"2021-08-21T15:01:18.026287+0000","last_undegraded":"2021-08-21T15:01:18.026287+0000","last_fullsized":"2021-08-21T15:01:18.026287+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":3859,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4013,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.027645+0000","last_change":"2021-08-21T15:01:18.027691+0000","last_active":"2021-08-21T15:00:52.413530+0000","last_peered":"2021-08-21T15:00:50.800815+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:00:22.745984+0000","last_became_peered":"2021-08-21T15:00:22.745984+0000","last_unstale":"2021-08-21T15:01:18.027645+0000","last_undegraded":"2021-08-21T15:01:18.027645+0000","last_fullsized":"2021-08-21T15:01:18.027645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":3986,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4006,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.030921+0000","last_change":"2021-08-21T15:01:18.030921+0000","last_active":"2021-08-21T15:00:52.843710+0000","last_peered":"2021-08-21T15:00:50.372465+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:00:22.746693+0000","last_became_peered":"2021-08-21T15:00:22.746693+0000","last_unstale":"2021-08-21T15:01:18.030921+0000","last_undegraded":"2021-08-21T15:01:18.030921+0000","last_fullsized":"2021-08-21T15:01:18.030921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":3988,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":3937,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.031224+0000","last_change":"2021-08-21T15:01:18.031261+0000","last_active":"2021-08-21T15:00:52.843632+0000","last_peered":"2021-08-21T15:00:50.696737+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:00:22.745452+0000","last_became_peered":"2021-08-21T15:00:22.745452+0000","last_unstale":"2021-08-21T15:01:18.031224+0000","last_undegraded":"2021-08-21T15:01:18.031224+0000","last_fullsized":"2021-08-21T15:01:18.031224+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":3923,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4063,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.026564+0000","last_change":"2021-08-21T15:01:18.026631+0000","last_active":"2021-08-21T15:00:52.409587+0000","last_peered":"2021-08-21T15:00:50.216409+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:00:22.748106+0000","last_became_peered":"2021-08-21T15:00:22.748106+0000","last_unstale":"2021-08-21T15:01:18.026564+0000","last_undegraded":"2021-08-21T15:01:18.026564+0000","last_fullsized":"2021-08-21T15:01:18.026564+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":4036,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4037,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.026735+0000","last_change":"2021-08-21T15:01:18.026735+0000","last_active":"2021-08-21T15:00:52.841845+0000","last_peered":"2021-08-21T15:00:50.856667+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:00:22.745861+0000","last_became_peered":"2021-08-21T15:00:22.745861+0000","last_unstale":"2021-08-21T15:01:18.026735+0000","last_undegraded":"2021-08-21T15:01:18.026735+0000","last_fullsized":"2021-08-21T15:01:18.026735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":4021,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":31665,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":259555328,"data_stored":411000,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":42949672978,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":42949672979,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672980,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20080,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":94351760,"statfs":{"total":96636764160,"available":96616202240,"internally_reserved":0,"allocated":172032,"data_stored":47445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.437}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.67000000000000004}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.61699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705685,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20080,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":94351760,"statfs":{"total":96636764160,"available":96616202240,"internally_reserved":0,"allocated":172032,"data_stored":47445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.35499999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:01:22.076 INFO:tasks.osd_recovery.ceph_manager:active or down! 2021-08-21T15:01:22.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:01:22.337 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:22.339 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:01:22.354 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":55,"stamp":"2021-08-21T15:01:21.462049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":31665,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":40160,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":39808,"kb_avail":188703520,"statfs":{"total":193273528320,"available":193232404480,"internally_reserved":0,"allocated":344064,"data_stored":94890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40763392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":31665,"num_objects_degraded":-27679,"num_objects_misplaced":-35651,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002248"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3973,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.029695+0000","last_change":"2021-08-21T15:01:18.029695+0000","last_active":"2021-08-21T15:00:52.843925+0000","last_peered":"2021-08-21T15:00:48.699483+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:00:22.747990+0000","last_became_peered":"2021-08-21T15:00:22.747990+0000","last_unstale":"2021-08-21T15:01:18.029695+0000","last_undegraded":"2021-08-21T15:01:18.029695+0000","last_fullsized":"2021-08-21T15:01:18.029695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":3956,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3923,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.028113+0000","last_change":"2021-08-21T15:01:18.028139+0000","last_active":"2021-08-21T15:00:52.409750+0000","last_peered":"2021-08-21T15:00:50.527252+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:00:22.747936+0000","last_became_peered":"2021-08-21T15:00:22.747936+0000","last_unstale":"2021-08-21T15:01:18.028113+0000","last_undegraded":"2021-08-21T15:01:18.028113+0000","last_fullsized":"2021-08-21T15:01:18.028113+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":3896,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3877,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.026287+0000","last_change":"2021-08-21T15:01:18.026329+0000","last_active":"2021-08-21T15:00:53.709677+0000","last_peered":"2021-08-21T15:00:52.399475+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:00:10.516390+0000","last_became_peered":"2021-08-21T15:00:10.516390+0000","last_unstale":"2021-08-21T15:01:18.026287+0000","last_undegraded":"2021-08-21T15:01:18.026287+0000","last_fullsized":"2021-08-21T15:01:18.026287+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":3859,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4013,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.027645+0000","last_change":"2021-08-21T15:01:18.027691+0000","last_active":"2021-08-21T15:00:52.413530+0000","last_peered":"2021-08-21T15:00:50.800815+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:00:22.745984+0000","last_became_peered":"2021-08-21T15:00:22.745984+0000","last_unstale":"2021-08-21T15:01:18.027645+0000","last_undegraded":"2021-08-21T15:01:18.027645+0000","last_fullsized":"2021-08-21T15:01:18.027645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":3986,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4006,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.030921+0000","last_change":"2021-08-21T15:01:18.030921+0000","last_active":"2021-08-21T15:00:52.843710+0000","last_peered":"2021-08-21T15:00:50.372465+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:00:22.746693+0000","last_became_peered":"2021-08-21T15:00:22.746693+0000","last_unstale":"2021-08-21T15:01:18.030921+0000","last_undegraded":"2021-08-21T15:01:18.030921+0000","last_fullsized":"2021-08-21T15:01:18.030921+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":3988,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":3937,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.031224+0000","last_change":"2021-08-21T15:01:18.031261+0000","last_active":"2021-08-21T15:00:52.843632+0000","last_peered":"2021-08-21T15:00:50.696737+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:00:22.745452+0000","last_became_peered":"2021-08-21T15:00:22.745452+0000","last_unstale":"2021-08-21T15:01:18.031224+0000","last_undegraded":"2021-08-21T15:01:18.031224+0000","last_fullsized":"2021-08-21T15:01:18.031224+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":3923,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4063,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.026564+0000","last_change":"2021-08-21T15:01:18.026631+0000","last_active":"2021-08-21T15:00:52.409587+0000","last_peered":"2021-08-21T15:00:50.216409+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:00:22.748106+0000","last_became_peered":"2021-08-21T15:00:22.748106+0000","last_unstale":"2021-08-21T15:01:18.026564+0000","last_undegraded":"2021-08-21T15:01:18.026564+0000","last_fullsized":"2021-08-21T15:01:18.026564+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":4036,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4037,"reported_epoch":20,"state":"down","last_fresh":"2021-08-21T15:01:18.026735+0000","last_change":"2021-08-21T15:01:18.026735+0000","last_active":"2021-08-21T15:00:52.841845+0000","last_peered":"2021-08-21T15:00:50.856667+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:00:22.745861+0000","last_became_peered":"2021-08-21T15:00:22.745861+0000","last_unstale":"2021-08-21T15:01:18.026735+0000","last_undegraded":"2021-08-21T15:01:18.026735+0000","last_fullsized":"2021-08-21T15:01:18.026735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":4021,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":31665,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":259555328,"data_stored":411000,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":42949672978,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":42949672979,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672980,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20080,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":94351760,"statfs":{"total":96636764160,"available":96616202240,"internally_reserved":0,"allocated":172032,"data_stored":47445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.437}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.67000000000000004}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.61699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705685,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20080,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":94351760,"statfs":{"total":96636764160,"available":96616202240,"internally_reserved":0,"allocated":172032,"data_stored":47445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.35499999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:01:22.355 INFO:tasks.ceph.osd.2:Restarting daemon 2021-08-21T15:01:22.356 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2021-08-21T15:01:22.358 INFO:tasks.ceph.osd.2:Started 2021-08-21T15:01:22.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2021-08-21T15:01:22.453 INFO:teuthology.orchestra.run.smithi165.stderr:no valid command found; 10 closest matches: 2021-08-21T15:01:22.453 INFO:teuthology.orchestra.run.smithi165.stderr:0 2021-08-21T15:01:22.453 INFO:teuthology.orchestra.run.smithi165.stderr:1 2021-08-21T15:01:22.453 INFO:teuthology.orchestra.run.smithi165.stderr:2 2021-08-21T15:01:22.454 INFO:teuthology.orchestra.run.smithi165.stderr:abort 2021-08-21T15:01:22.454 INFO:teuthology.orchestra.run.smithi165.stderr:assert 2021-08-21T15:01:22.454 INFO:teuthology.orchestra.run.smithi165.stderr:config diff 2021-08-21T15:01:22.454 INFO:teuthology.orchestra.run.smithi165.stderr:config diff get 2021-08-21T15:01:22.455 INFO:teuthology.orchestra.run.smithi165.stderr:config get 2021-08-21T15:01:22.455 INFO:teuthology.orchestra.run.smithi165.stderr:config help [] 2021-08-21T15:01:22.455 INFO:teuthology.orchestra.run.smithi165.stderr:config set ... 2021-08-21T15:01:22.455 INFO:teuthology.orchestra.run.smithi165.stderr:admin_socket: invalid command 2021-08-21T15:01:22.456 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-08-21T15:01:22.457 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2021-08-21T15:01:22.978 INFO:tasks.ceph.osd.2.smithi165.stderr:2021-08-21T15:01:22.975+0000 7f5d750e0f40 -1 Falling back to public interface 2021-08-21T15:01:26.870 INFO:tasks.ceph.osd.2.smithi165.stderr:2021-08-21T15:01:26.867+0000 7f5d750e0f40 -1 bluestore::NCB::restore_allocator::No Valid allocation info on disk (empty file) 2021-08-21T15:01:27.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2021-08-21T15:01:27.552 INFO:teuthology.orchestra.run.smithi165.stderr:no valid command found; 10 closest matches: 2021-08-21T15:01:27.552 INFO:teuthology.orchestra.run.smithi165.stderr:0 2021-08-21T15:01:27.553 INFO:teuthology.orchestra.run.smithi165.stderr:1 2021-08-21T15:01:27.553 INFO:teuthology.orchestra.run.smithi165.stderr:2 2021-08-21T15:01:27.553 INFO:teuthology.orchestra.run.smithi165.stderr:abort 2021-08-21T15:01:27.553 INFO:teuthology.orchestra.run.smithi165.stderr:assert 2021-08-21T15:01:27.554 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs debug_inject_read_zeros 2021-08-21T15:01:27.554 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs files list 2021-08-21T15:01:27.554 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs stats 2021-08-21T15:01:27.554 INFO:teuthology.orchestra.run.smithi165.stderr:bluestore allocator dump block 2021-08-21T15:01:27.555 INFO:teuthology.orchestra.run.smithi165.stderr:bluestore allocator fragmentation block 2021-08-21T15:01:27.555 INFO:teuthology.orchestra.run.smithi165.stderr:admin_socket: invalid command 2021-08-21T15:01:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-08-21T15:01:27.556 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2021-08-21T15:01:30.422 INFO:tasks.ceph.osd.2.smithi165.stderr:2021-08-21T15:01:30.419+0000 7f5d750e0f40 -1 bluestore::NCB::restore_allocator::No Valid allocation info on disk (empty file) 2021-08-21T15:01:32.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2021-08-21T15:01:32.693 INFO:teuthology.orchestra.run.smithi165.stderr:no valid command found; 10 closest matches: 2021-08-21T15:01:32.693 INFO:teuthology.orchestra.run.smithi165.stderr:0 2021-08-21T15:01:32.694 INFO:teuthology.orchestra.run.smithi165.stderr:1 2021-08-21T15:01:32.694 INFO:teuthology.orchestra.run.smithi165.stderr:2 2021-08-21T15:01:32.694 INFO:teuthology.orchestra.run.smithi165.stderr:abort 2021-08-21T15:01:32.694 INFO:teuthology.orchestra.run.smithi165.stderr:assert 2021-08-21T15:01:32.695 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs debug_inject_read_zeros 2021-08-21T15:01:32.695 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs files list 2021-08-21T15:01:32.695 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs stats 2021-08-21T15:01:32.695 INFO:teuthology.orchestra.run.smithi165.stderr:bluestore allocator dump block 2021-08-21T15:01:32.695 INFO:teuthology.orchestra.run.smithi165.stderr:bluestore allocator fragmentation block 2021-08-21T15:01:32.696 INFO:teuthology.orchestra.run.smithi165.stderr:admin_socket: invalid command 2021-08-21T15:01:32.696 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-08-21T15:01:32.697 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2021-08-21T15:01:33.983 INFO:tasks.ceph.osd.2.smithi165.stderr:2021-08-21T15:01:33.979+0000 7f5d750e0f40 -1 osd.2 19 log_to_monitors {default=true} 2021-08-21T15:01:34.731 INFO:tasks.ceph.osd.2.smithi165.stderr:2021-08-21T15:01:34.731+0000 7f5d6a627700 -1 osd.2 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-08-21T15:01:37.699 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2021-08-21T15:01:37.834 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2021-08-21T15:01:37.835 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2021-08-21T15:01:37.835 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2021-08-21T15:01:37.835 INFO:teuthology.orchestra.run.smithi165.stdout:} 2021-08-21T15:01:37.846 INFO:tasks.ceph.osd.3:Restarting daemon 2021-08-21T15:01:37.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2021-08-21T15:01:37.850 INFO:tasks.ceph.osd.3:Started 2021-08-21T15:01:37.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2021-08-21T15:01:37.954 INFO:teuthology.orchestra.run.smithi165.stderr:no valid command found; 10 closest matches: 2021-08-21T15:01:37.955 INFO:teuthology.orchestra.run.smithi165.stderr:0 2021-08-21T15:01:37.955 INFO:teuthology.orchestra.run.smithi165.stderr:1 2021-08-21T15:01:37.955 INFO:teuthology.orchestra.run.smithi165.stderr:2 2021-08-21T15:01:37.956 INFO:teuthology.orchestra.run.smithi165.stderr:abort 2021-08-21T15:01:37.956 INFO:teuthology.orchestra.run.smithi165.stderr:assert 2021-08-21T15:01:37.956 INFO:teuthology.orchestra.run.smithi165.stderr:config diff 2021-08-21T15:01:37.956 INFO:teuthology.orchestra.run.smithi165.stderr:config diff get 2021-08-21T15:01:37.957 INFO:teuthology.orchestra.run.smithi165.stderr:config get 2021-08-21T15:01:37.957 INFO:teuthology.orchestra.run.smithi165.stderr:config help [] 2021-08-21T15:01:37.957 INFO:teuthology.orchestra.run.smithi165.stderr:config set ... 2021-08-21T15:01:37.958 INFO:teuthology.orchestra.run.smithi165.stderr:admin_socket: invalid command 2021-08-21T15:01:37.959 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-08-21T15:01:37.960 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2021-08-21T15:01:38.477 INFO:tasks.ceph.osd.3.smithi165.stderr:2021-08-21T15:01:38.475+0000 7ff4e2a8ef40 -1 Falling back to public interface 2021-08-21T15:01:41.992 INFO:tasks.ceph.osd.3.smithi165.stderr:2021-08-21T15:01:41.991+0000 7ff4e2a8ef40 -1 bluestore::NCB::restore_allocator::No Valid allocation info on disk (empty file) 2021-08-21T15:01:42.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2021-08-21T15:01:43.058 INFO:teuthology.orchestra.run.smithi165.stderr:no valid command found; 10 closest matches: 2021-08-21T15:01:43.058 INFO:teuthology.orchestra.run.smithi165.stderr:0 2021-08-21T15:01:43.058 INFO:teuthology.orchestra.run.smithi165.stderr:1 2021-08-21T15:01:43.059 INFO:teuthology.orchestra.run.smithi165.stderr:2 2021-08-21T15:01:43.059 INFO:teuthology.orchestra.run.smithi165.stderr:abort 2021-08-21T15:01:43.059 INFO:teuthology.orchestra.run.smithi165.stderr:assert 2021-08-21T15:01:43.060 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs debug_inject_read_zeros 2021-08-21T15:01:43.060 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs files list 2021-08-21T15:01:43.060 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs stats 2021-08-21T15:01:43.060 INFO:teuthology.orchestra.run.smithi165.stderr:bluestore allocator dump block 2021-08-21T15:01:43.060 INFO:teuthology.orchestra.run.smithi165.stderr:bluestore allocator fragmentation block 2021-08-21T15:01:43.061 INFO:teuthology.orchestra.run.smithi165.stderr:admin_socket: invalid command 2021-08-21T15:01:43.061 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-08-21T15:01:43.062 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2021-08-21T15:01:45.709 INFO:tasks.ceph.osd.3.smithi165.stderr:2021-08-21T15:01:45.707+0000 7ff4e2a8ef40 -1 bluestore::NCB::restore_allocator::No Valid allocation info on disk (empty file) 2021-08-21T15:01:48.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2021-08-21T15:01:48.169 INFO:teuthology.orchestra.run.smithi165.stderr:no valid command found; 10 closest matches: 2021-08-21T15:01:48.170 INFO:teuthology.orchestra.run.smithi165.stderr:0 2021-08-21T15:01:48.170 INFO:teuthology.orchestra.run.smithi165.stderr:1 2021-08-21T15:01:48.170 INFO:teuthology.orchestra.run.smithi165.stderr:2 2021-08-21T15:01:48.171 INFO:teuthology.orchestra.run.smithi165.stderr:abort 2021-08-21T15:01:48.171 INFO:teuthology.orchestra.run.smithi165.stderr:assert 2021-08-21T15:01:48.171 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs debug_inject_read_zeros 2021-08-21T15:01:48.172 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs files list 2021-08-21T15:01:48.172 INFO:teuthology.orchestra.run.smithi165.stderr:bluefs stats 2021-08-21T15:01:48.173 INFO:teuthology.orchestra.run.smithi165.stderr:bluestore allocator dump block 2021-08-21T15:01:48.176 INFO:teuthology.orchestra.run.smithi165.stderr:bluestore allocator fragmentation block 2021-08-21T15:01:48.176 INFO:teuthology.orchestra.run.smithi165.stderr:admin_socket: invalid command 2021-08-21T15:01:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-08-21T15:01:48.179 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2021-08-21T15:01:49.235 INFO:tasks.ceph.osd.3.smithi165.stderr:2021-08-21T15:01:49.235+0000 7ff4e2a8ef40 -1 osd.3 19 log_to_monitors {default=true} 2021-08-21T15:01:49.851 INFO:tasks.ceph.osd.3.smithi165.stderr:2021-08-21T15:01:49.851+0000 7ff4dafdb700 -1 osd.3 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-08-21T15:01:53.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2021-08-21T15:01:53.279 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2021-08-21T15:01:53.279 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2021-08-21T15:01:53.279 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2021-08-21T15:01:53.279 INFO:teuthology.orchestra.run.smithi165.stdout:} 2021-08-21T15:01:53.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2021-08-21T15:01:53.561 INFO:teuthology.orchestra.run.smithi165.stdout:epoch 29 2021-08-21T15:01:53.561 INFO:teuthology.orchestra.run.smithi165.stdout:fsid b82e8492-200f-4d81-953f-d1ffd38e4baa 2021-08-21T15:01:53.561 INFO:teuthology.orchestra.run.smithi165.stdout:created 2021-08-21T14:59:59.722740+0000 2021-08-21T15:01:53.561 INFO:teuthology.orchestra.run.smithi165.stdout:modified 2021-08-21T15:01:52.874542+0000 2021-08-21T15:01:53.562 INFO:teuthology.orchestra.run.smithi165.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2021-08-21T15:01:53.562 INFO:teuthology.orchestra.run.smithi165.stdout:crush_version 7 2021-08-21T15:01:53.562 INFO:teuthology.orchestra.run.smithi165.stdout:full_ratio 0.95 2021-08-21T15:01:53.562 INFO:teuthology.orchestra.run.smithi165.stdout:backfillfull_ratio 0.9 2021-08-21T15:01:53.563 INFO:teuthology.orchestra.run.smithi165.stdout:nearfull_ratio 0.85 2021-08-21T15:01:53.563 INFO:teuthology.orchestra.run.smithi165.stdout:require_min_compat_client luminous 2021-08-21T15:01:53.563 INFO:teuthology.orchestra.run.smithi165.stdout:min_compat_client jewel 2021-08-21T15:01:53.563 INFO:teuthology.orchestra.run.smithi165.stdout:require_osd_release quincy 2021-08-21T15:01:53.564 INFO:teuthology.orchestra.run.smithi165.stdout:stretch_mode_enabled false 2021-08-21T15:01:53.564 INFO:teuthology.orchestra.run.smithi165.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 13 flags hashpspool stripe_width 0 application rbd 2021-08-21T15:01:53.564 INFO:teuthology.orchestra.run.smithi165.stdout:max_osd 4 2021-08-21T15:01:53.564 INFO:teuthology.orchestra.run.smithi165.stdout:osd.0 up in weight 1 up_from 9 up_thru 27 down_at 0 last_clean_interval [0,0) v2:172.21.15.165:6808/16490 v2:172.21.15.165:6809/16490 exists,up f6dc722a-fc2c-46bd-ade5-b15534b9209a 2021-08-21T15:01:53.564 INFO:teuthology.orchestra.run.smithi165.stdout:osd.1 up in weight 1 up_from 10 up_thru 23 down_at 0 last_clean_interval [0,0) v2:172.21.15.165:6804/16500 v2:172.21.15.165:6805/16500 exists,up a8ed204d-2f11-474e-85a1-d842c1c86929 2021-08-21T15:01:53.565 INFO:teuthology.orchestra.run.smithi165.stdout:osd.2 up out weight 0 up_from 23 up_thru 24 down_at 20 last_clean_interval [10,19) v2:172.21.15.165:6800/22293 v2:172.21.15.165:6801/22293 exists,up f47a338b-b210-48d7-9b2b-d2bc504ad371 2021-08-21T15:01:53.565 INFO:teuthology.orchestra.run.smithi165.stdout:osd.3 up out weight 0 up_from 27 up_thru 28 down_at 20 last_clean_interval [10,19) v2:172.21.15.165:6812/23145 v2:172.21.15.165:6813/23145 exists,up 6ece6914-c949-456a-99db-3479cfa37199 2021-08-21T15:01:53.565 INFO:teuthology.orchestra.run.smithi165.stdout:pg_temp 1.0 [2] 2021-08-21T15:01:53.565 INFO:teuthology.orchestra.run.smithi165.stdout:pg_temp 1.1 [3] 2021-08-21T15:01:53.565 INFO:teuthology.orchestra.run.smithi165.stdout:pg_temp 1.2 [2] 2021-08-21T15:01:53.566 INFO:teuthology.orchestra.run.smithi165.stdout:pg_temp 1.3 [2] 2021-08-21T15:01:53.566 INFO:teuthology.orchestra.run.smithi165.stdout:pg_temp 1.4 [3] 2021-08-21T15:01:53.566 INFO:teuthology.orchestra.run.smithi165.stdout:pg_temp 1.5 [2] 2021-08-21T15:01:53.566 INFO:teuthology.orchestra.run.smithi165.stdout:pg_temp 1.6 [3] 2021-08-21T15:01:53.567 INFO:teuthology.orchestra.run.smithi165.stdout:pg_temp 1.7 [2] 2021-08-21T15:01:53.572 INFO:tasks.osd_recovery.ceph_manager:['osd.0 up in weight 1 up_from 9 up_thru 27 down_at 0 last_clean_interval [0,0) v2:172.21.15.165:6808/16490 v2:172.21.15.165:6809/16490 exists,up f6dc722a-fc2c-46bd-ade5-b15534b9209a', 'osd.1 up in weight 1 up_from 10 up_thru 23 down_at 0 last_clean_interval [0,0) v2:172.21.15.165:6804/16500 v2:172.21.15.165:6805/16500 exists,up a8ed204d-2f11-474e-85a1-d842c1c86929', 'osd.2 up out weight 0 up_from 23 up_thru 24 down_at 20 last_clean_interval [10,19) v2:172.21.15.165:6800/22293 v2:172.21.15.165:6801/22293 exists,up f47a338b-b210-48d7-9b2b-d2bc504ad371', 'osd.3 up out weight 0 up_from 27 up_thru 28 down_at 20 last_clean_interval [10,19) v2:172.21.15.165:6812/23145 v2:172.21.15.165:6813/23145 exists,up 6ece6914-c949-456a-99db-3479cfa37199'] 2021-08-21T15:01:53.573 INFO:tasks.osd_recovery:all are up! 2021-08-21T15:01:53.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 0 2021-08-21T15:01:53.711 INFO:teuthology.orchestra.run.smithi165.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-08-21T15:01:53.721 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 debug kick_recovery_wq 0 2021-08-21T15:01:53.854 INFO:teuthology.orchestra.run.smithi165.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-08-21T15:01:53.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 0 2021-08-21T15:01:53.976 INFO:teuthology.orchestra.run.smithi165.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-08-21T15:01:53.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 debug kick_recovery_wq 0 2021-08-21T15:01:54.119 INFO:teuthology.orchestra.run.smithi165.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-08-21T15:01:54.129 INFO:tasks.osd_recovery.ceph_manager:waiting for clean 2021-08-21T15:01:54.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:01:54.387 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:54.390 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:01:54.399 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2021-08-21T15:01:53.471093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":11918,"num_objects_degraded":19625,"num_objects_misplaced":19625,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":41640,"kb_used_data":1304,"kb_used_omap":0,"kb_used_meta":40320,"kb_avail":188702040,"statfs":{"total":193273528320,"available":193230888960,"internally_reserved":0,"allocated":1335296,"data_stored":123918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41287680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-52,"num_objects_misplaced":-52,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001311"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3984,"reported_epoch":25,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:37.910922+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:37.910922+0000","last_peered":"2021-08-21T15:01:37.910922+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:37.910922+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3929,"reported_epoch":24,"state":"down","last_fresh":"2021-08-21T15:01:36.748204+0000","last_change":"2021-08-21T15:01:35.735890+0000","last_active":"2021-08-21T15:00:52.409750+0000","last_peered":"2021-08-21T15:00:50.527252+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:00:22.747936+0000","last_became_peered":"2021-08-21T15:00:22.747936+0000","last_unstale":"2021-08-21T15:01:36.748204+0000","last_undegraded":"2021-08-21T15:01:36.748204+0000","last_fullsized":"2021-08-21T15:01:36.748204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":3896,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3889,"reported_epoch":25,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:37.917042+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:37.917042+0000","last_peered":"2021-08-21T15:01:37.917042+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:37.917042+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4019,"reported_epoch":24,"state":"down","last_fresh":"2021-08-21T15:01:36.747762+0000","last_change":"2021-08-21T15:01:35.735891+0000","last_active":"2021-08-21T15:00:52.413530+0000","last_peered":"2021-08-21T15:00:50.800815+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:00:22.745984+0000","last_became_peered":"2021-08-21T15:00:22.745984+0000","last_unstale":"2021-08-21T15:01:36.747762+0000","last_undegraded":"2021-08-21T15:01:36.747762+0000","last_fullsized":"2021-08-21T15:01:36.747762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":3986,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4018,"reported_epoch":25,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:37.930137+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:37.930137+0000","last_peered":"2021-08-21T15:01:37.930137+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:37.930137+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":4317,"reported_epoch":25,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:49.727678+0000","last_change":"2021-08-21T15:01:37.899328+0000","last_active":"2021-08-21T15:01:49.727678+0000","last_peered":"2021-08-21T15:01:49.727678+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:01:49.727678+0000","last_undegraded":"2021-08-21T15:01:37.879628+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3801,"num_objects_misplaced":3801,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":122}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4069,"reported_epoch":24,"state":"down","last_fresh":"2021-08-21T15:01:36.747704+0000","last_change":"2021-08-21T15:01:35.735872+0000","last_active":"2021-08-21T15:00:52.409587+0000","last_peered":"2021-08-21T15:00:50.216409+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:00:22.748106+0000","last_became_peered":"2021-08-21T15:00:22.748106+0000","last_unstale":"2021-08-21T15:01:36.747704+0000","last_undegraded":"2021-08-21T15:01:36.747704+0000","last_fullsized":"2021-08-21T15:01:36.747704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":4036,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4049,"reported_epoch":25,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:37.898051+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:37.898051+0000","last_peered":"2021-08-21T15:01:37.898051+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:37.898051+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":11918,"num_objects_degraded":19625,"num_objects_misplaced":19625,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":260489216,"data_stored":411228,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":1,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247812,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672986,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20832,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":20160,"kb_avail":94351008,"statfs":{"total":96636764160,"available":96615432192,"internally_reserved":0,"allocated":679936,"data_stored":61962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20643840},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.248},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.33900000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.371},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.29299999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.61699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20808,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":20160,"kb_avail":94351032,"statfs":{"total":96636764160,"available":96615456768,"internally_reserved":0,"allocated":655360,"data_stored":61956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20643840},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.41999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.35899999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:01:54.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:01:54.658 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:54.661 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:01:54.670 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2021-08-21T15:01:53.471093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":11918,"num_objects_degraded":19625,"num_objects_misplaced":19625,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":41640,"kb_used_data":1304,"kb_used_omap":0,"kb_used_meta":40320,"kb_avail":188702040,"statfs":{"total":193273528320,"available":193230888960,"internally_reserved":0,"allocated":1335296,"data_stored":123918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41287680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-52,"num_objects_misplaced":-52,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001311"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3984,"reported_epoch":25,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:37.910922+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:37.910922+0000","last_peered":"2021-08-21T15:01:37.910922+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:37.910922+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3929,"reported_epoch":24,"state":"down","last_fresh":"2021-08-21T15:01:36.748204+0000","last_change":"2021-08-21T15:01:35.735890+0000","last_active":"2021-08-21T15:00:52.409750+0000","last_peered":"2021-08-21T15:00:50.527252+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:00:22.747936+0000","last_became_peered":"2021-08-21T15:00:22.747936+0000","last_unstale":"2021-08-21T15:01:36.748204+0000","last_undegraded":"2021-08-21T15:01:36.748204+0000","last_fullsized":"2021-08-21T15:01:36.748204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":3896,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3889,"reported_epoch":25,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:37.917042+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:37.917042+0000","last_peered":"2021-08-21T15:01:37.917042+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:37.917042+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4019,"reported_epoch":24,"state":"down","last_fresh":"2021-08-21T15:01:36.747762+0000","last_change":"2021-08-21T15:01:35.735891+0000","last_active":"2021-08-21T15:00:52.413530+0000","last_peered":"2021-08-21T15:00:50.800815+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:00:22.745984+0000","last_became_peered":"2021-08-21T15:00:22.745984+0000","last_unstale":"2021-08-21T15:01:36.747762+0000","last_undegraded":"2021-08-21T15:01:36.747762+0000","last_fullsized":"2021-08-21T15:01:36.747762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":3986,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4018,"reported_epoch":25,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:37.930137+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:37.930137+0000","last_peered":"2021-08-21T15:01:37.930137+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:37.930137+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":4317,"reported_epoch":25,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:49.727678+0000","last_change":"2021-08-21T15:01:37.899328+0000","last_active":"2021-08-21T15:01:49.727678+0000","last_peered":"2021-08-21T15:01:49.727678+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:01:49.727678+0000","last_undegraded":"2021-08-21T15:01:37.879628+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3801,"num_objects_misplaced":3801,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":122}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4069,"reported_epoch":24,"state":"down","last_fresh":"2021-08-21T15:01:36.747704+0000","last_change":"2021-08-21T15:01:35.735872+0000","last_active":"2021-08-21T15:00:52.409587+0000","last_peered":"2021-08-21T15:00:50.216409+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:00:22.748106+0000","last_became_peered":"2021-08-21T15:00:22.748106+0000","last_unstale":"2021-08-21T15:01:36.747704+0000","last_undegraded":"2021-08-21T15:01:36.747704+0000","last_fullsized":"2021-08-21T15:01:36.747704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":4036,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4049,"reported_epoch":25,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:37.898051+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:37.898051+0000","last_peered":"2021-08-21T15:01:37.898051+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:37.898051+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":11918,"num_objects_degraded":19625,"num_objects_misplaced":19625,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":260489216,"data_stored":411228,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":1,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247812,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672986,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20832,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":20160,"kb_avail":94351008,"statfs":{"total":96636764160,"available":96615432192,"internally_reserved":0,"allocated":679936,"data_stored":61962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20643840},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.248},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.33900000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.371},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.29299999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.61699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20808,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":20160,"kb_avail":94351032,"statfs":{"total":96636764160,"available":96615456768,"internally_reserved":0,"allocated":655360,"data_stored":61956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20643840},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.41999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.35899999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:01:54.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:01:54.984 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:54.985 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 3 pgs inactive, 3 pgs down","count":6},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19625/63330 objects degraded (30.988%), 5 pgs degraded","count":5},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":115,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"down","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":42639360,"bytes_avail":193230888960,"bytes_total":193273528320,"inactive_pgs_ratio":0.375,"degraded_objects":19625,"degraded_total":63330,"degraded_ratio":0.30988473077530398,"misplaced_objects":19625,"misplaced_total":63330,"misplaced_ratio":0.30988473077530398,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":12,"recovering_keys_per_sec":0,"num_objects_recovered":52,"num_bytes_recovered":52,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2021-08-21T15:01:11.459247+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (55s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (55s)\n [............................] (remaining: 9h)","progress":0.00045006527216173708,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (55s)\n [............................] (remaining: 9h)","progress":0.00045006527216173708,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (50s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (55s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:01:54.996 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:01:54.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:01:55.255 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:55.259 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:01:55.268 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2021-08-21T15:01:53.471093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":11918,"num_objects_degraded":19625,"num_objects_misplaced":19625,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":41640,"kb_used_data":1304,"kb_used_omap":0,"kb_used_meta":40320,"kb_avail":188702040,"statfs":{"total":193273528320,"available":193230888960,"internally_reserved":0,"allocated":1335296,"data_stored":123918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41287680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-52,"num_objects_misplaced":-52,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001311"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3984,"reported_epoch":25,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:37.910922+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:37.910922+0000","last_peered":"2021-08-21T15:01:37.910922+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:37.910922+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3929,"reported_epoch":24,"state":"down","last_fresh":"2021-08-21T15:01:36.748204+0000","last_change":"2021-08-21T15:01:35.735890+0000","last_active":"2021-08-21T15:00:52.409750+0000","last_peered":"2021-08-21T15:00:50.527252+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:00:22.747936+0000","last_became_peered":"2021-08-21T15:00:22.747936+0000","last_unstale":"2021-08-21T15:01:36.748204+0000","last_undegraded":"2021-08-21T15:01:36.748204+0000","last_fullsized":"2021-08-21T15:01:36.748204+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":3896,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3889,"reported_epoch":25,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:37.917042+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:37.917042+0000","last_peered":"2021-08-21T15:01:37.917042+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:37.917042+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4019,"reported_epoch":24,"state":"down","last_fresh":"2021-08-21T15:01:36.747762+0000","last_change":"2021-08-21T15:01:35.735891+0000","last_active":"2021-08-21T15:00:52.413530+0000","last_peered":"2021-08-21T15:00:50.800815+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:00:22.745984+0000","last_became_peered":"2021-08-21T15:00:22.745984+0000","last_unstale":"2021-08-21T15:01:36.747762+0000","last_undegraded":"2021-08-21T15:01:36.747762+0000","last_fullsized":"2021-08-21T15:01:36.747762+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":3986,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4018,"reported_epoch":25,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:37.930137+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:37.930137+0000","last_peered":"2021-08-21T15:01:37.930137+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:37.930137+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":4317,"reported_epoch":25,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:49.727678+0000","last_change":"2021-08-21T15:01:37.899328+0000","last_active":"2021-08-21T15:01:49.727678+0000","last_peered":"2021-08-21T15:01:49.727678+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:01:49.727678+0000","last_undegraded":"2021-08-21T15:01:37.879628+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3801,"num_objects_misplaced":3801,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":122}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4069,"reported_epoch":24,"state":"down","last_fresh":"2021-08-21T15:01:36.747704+0000","last_change":"2021-08-21T15:01:35.735872+0000","last_active":"2021-08-21T15:00:52.409587+0000","last_peered":"2021-08-21T15:00:50.216409+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:00:22.748106+0000","last_became_peered":"2021-08-21T15:00:22.748106+0000","last_unstale":"2021-08-21T15:01:36.747704+0000","last_undegraded":"2021-08-21T15:01:36.747704+0000","last_fullsized":"2021-08-21T15:01:36.747704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":4036,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4049,"reported_epoch":25,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:37.898051+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:37.898051+0000","last_peered":"2021-08-21T15:01:37.898051+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:37.898051+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":11918,"num_objects_degraded":19625,"num_objects_misplaced":19625,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":260489216,"data_stored":411228,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":1,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247812,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672986,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20832,"kb_used_data":664,"kb_used_omap":0,"kb_used_meta":20160,"kb_avail":94351008,"statfs":{"total":96636764160,"available":96615432192,"internally_reserved":0,"allocated":679936,"data_stored":61962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20643840},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.248},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.33900000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.371},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.29299999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.61699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":20808,"kb_used_data":640,"kb_used_omap":0,"kb_used_meta":20160,"kb_avail":94351032,"statfs":{"total":96636764160,"available":96615456768,"internally_reserved":0,"allocated":655360,"data_stored":61956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20643840},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.41999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.35899999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":446464,"data_stored":109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:01:58.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:01:58.535 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:58.538 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:01:58.552 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2021-08-21T15:01:57.472345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35242,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":173,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":42368,"kb_used_data":1776,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":188701312,"statfs":{"total":193273528320,"available":193230143488,"internally_reserved":0,"allocated":1818624,"data_stored":147932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-11918,"num_objects_degraded":8117,"num_objects_misplaced":15617,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":51,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.623149"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":4474,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:01:54.726881+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:01:54.726881+0000","last_peered":"2021-08-21T15:01:54.726881+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:01:54.726881+0000","last_undegraded":"2021-08-21T15:01:54.726881+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7500,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":173,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":173},{"shards":"2,3","objects":3750}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35242,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":173,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":260907008,"data_stored":411330,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964116994,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247813,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672987,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21196,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":20288,"kb_avail":94350644,"statfs":{"total":96636764160,"available":96615059456,"internally_reserved":0,"allocated":921600,"data_stored":73810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20774912},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.496},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.61699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21172,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":20288,"kb_avail":94350668,"statfs":{"total":96636764160,"available":96615084032,"internally_reserved":0,"allocated":897024,"data_stored":74122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20774912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.52900000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.81899999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.88}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":696320,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:01:58.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:01:58.874 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:58.874 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":119,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":43384832,"bytes_avail":193230143488,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":35242,"misplaced_total":63330,"misplaced_ratio":0.5564819201010579,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":7,"recovering_keys_per_sec":0,"num_objects_recovered":51,"num_bytes_recovered":51,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2021-08-21T15:01:11.459247+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (60s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (60s)\n [............................] (remaining: 21h)","progress":0.00078439945355057716,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (60s)\n [............................] (remaining: 21h)","progress":0.00078439945355057716,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (55s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (60s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:01:58.890 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:01:58.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:01:59.156 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:01:59.159 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:01:59.169 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2021-08-21T15:01:57.472345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35242,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":173,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":42368,"kb_used_data":1776,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":188701312,"statfs":{"total":193273528320,"available":193230143488,"internally_reserved":0,"allocated":1818624,"data_stored":147932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-11918,"num_objects_degraded":8117,"num_objects_misplaced":15617,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":51,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.623149"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":4474,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:01:54.726881+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:01:54.726881+0000","last_peered":"2021-08-21T15:01:54.726881+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:01:54.726881+0000","last_undegraded":"2021-08-21T15:01:54.726881+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7500,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":173,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":173},{"shards":"2,3","objects":3750}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35242,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":173,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":260907008,"data_stored":411330,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964116994,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247813,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672987,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21196,"kb_used_data":900,"kb_used_omap":0,"kb_used_meta":20288,"kb_avail":94350644,"statfs":{"total":96636764160,"available":96615059456,"internally_reserved":0,"allocated":921600,"data_stored":73810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20774912},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.496},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.61699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21172,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":20288,"kb_avail":94350668,"statfs":{"total":96636764160,"available":96615084032,"internally_reserved":0,"allocated":897024,"data_stored":74122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20774912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.52900000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.81899999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.88}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":696320,"data_stored":170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:02.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:02.437 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:02.440 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:02.450 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2021-08-21T15:02:01.473545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35140,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":42896,"kb_used_data":2176,"kb_used_omap":0,"kb_used_meta":40704,"kb_avail":188700784,"statfs":{"total":193273528320,"available":193229602816,"internally_reserved":0,"allocated":2228224,"data_stored":148032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41680896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-11918,"num_objects_degraded":8117,"num_objects_misplaced":15515,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.593817"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":4627,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:01:59.729079+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:01:59.729079+0000","last_peered":"2021-08-21T15:01:59.729079+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:01:59.729079+0000","last_undegraded":"2021-08-21T15:01:59.729079+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7398,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":224},{"shards":"2,3","objects":3699}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35140,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":261324800,"data_stored":411432,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964116995,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672988,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21460,"kb_used_data":1100,"kb_used_omap":0,"kb_used_meta":20352,"kb_avail":94350380,"statfs":{"total":96636764160,"available":96614789120,"internally_reserved":0,"allocated":1126400,"data_stored":73860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20840448},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.46700000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.439},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.35299999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.58799999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705693,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21436,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":20352,"kb_avail":94350404,"statfs":{"total":96636764160,"available":96614813696,"internally_reserved":0,"allocated":1101824,"data_stored":74172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20840448},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.50900000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.58199999999999996}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.432},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":864256,"data_stored":211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":905216,"data_stored":221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:02.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:02.762 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:02.762 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":123,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":43925504,"bytes_avail":193229602816,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":35140,"misplaced_total":63330,"misplaced_ratio":0.55487130901626402,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":10,"recovering_keys_per_sec":0,"num_objects_recovered":102,"num_bytes_recovered":102,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2021-08-21T15:01:11.459247+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (65s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (65s)\n [............................] (remaining: 16h)","progress":0.0011123041622340679,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (65s)\n [............................] (remaining: 16h)","progress":0.0011123041622340679,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (60s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (65s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:02.779 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:02.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:03.047 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:03.049 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:03.059 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2021-08-21T15:02:01.473545+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35140,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":42896,"kb_used_data":2176,"kb_used_omap":0,"kb_used_meta":40704,"kb_avail":188700784,"statfs":{"total":193273528320,"available":193229602816,"internally_reserved":0,"allocated":2228224,"data_stored":148032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41680896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-11918,"num_objects_degraded":8117,"num_objects_misplaced":15515,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.593817"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":4627,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:01:59.729079+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:01:59.729079+0000","last_peered":"2021-08-21T15:01:59.729079+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:01:59.729079+0000","last_undegraded":"2021-08-21T15:01:59.729079+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7398,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":224},{"shards":"2,3","objects":3699}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35140,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":261324800,"data_stored":411432,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964116995,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672988,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21460,"kb_used_data":1100,"kb_used_omap":0,"kb_used_meta":20352,"kb_avail":94350380,"statfs":{"total":96636764160,"available":96614789120,"internally_reserved":0,"allocated":1126400,"data_stored":73860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20840448},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.46700000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.439},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.35299999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.58799999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705693,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21436,"kb_used_data":1076,"kb_used_omap":0,"kb_used_meta":20352,"kb_avail":94350404,"statfs":{"total":96636764160,"available":96614813696,"internally_reserved":0,"allocated":1101824,"data_stored":74172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20840448},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.50900000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.58199999999999996}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.432},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":864256,"data_stored":211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":905216,"data_stored":221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:06.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:06.324 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:06.326 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:06.336 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2021-08-21T15:02:05.474723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35038,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":43496,"kb_used_data":2584,"kb_used_omap":0,"kb_used_meta":40896,"kb_avail":188700184,"statfs":{"total":193273528320,"available":193228988416,"internally_reserved":0,"allocated":2646016,"data_stored":148134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41877504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-11918,"num_objects_degraded":8117,"num_objects_misplaced":15413,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003629"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":4780,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:04.700665+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:04.700665+0000","last_peered":"2021-08-21T15:02:04.700665+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:04.700665+0000","last_undegraded":"2021-08-21T15:02:04.700665+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7296,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":275},{"shards":"2,3","objects":3648}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35038,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":261742592,"data_stored":411534,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964116996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247815,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672989,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21724,"kb_used_data":1300,"kb_used_omap":0,"kb_used_meta":20416,"kb_avail":94350116,"statfs":{"total":96636764160,"available":96614518784,"internally_reserved":0,"allocated":1331200,"data_stored":73910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20905984},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.58499999999999996}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.50700000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.54300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21772,"kb_used_data":1284,"kb_used_omap":0,"kb_used_meta":20480,"kb_avail":94350068,"statfs":{"total":96636764160,"available":96614469632,"internally_reserved":0,"allocated":1314816,"data_stored":74224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20971520},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.317},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.51400000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.56699999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1114112,"data_stored":272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:06.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:06.642 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:06.642 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":126,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":44539904,"bytes_avail":193228988416,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":35038,"misplaced_total":63330,"misplaced_ratio":0.55326069793147004,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":12,"recovering_keys_per_sec":0,"num_objects_recovered":153,"num_bytes_recovered":153,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2021-08-21T15:01:11.459247+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (70s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (70s)\n [............................] (remaining: 13h)","progress":0.0014402088709175587,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (70s)\n [............................] (remaining: 13h)","progress":0.0014402088709175587,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (65s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (70s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:06.655 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:06.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:06.894 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:06.897 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:06.906 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2021-08-21T15:02:05.474723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35038,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":43496,"kb_used_data":2584,"kb_used_omap":0,"kb_used_meta":40896,"kb_avail":188700184,"statfs":{"total":193273528320,"available":193228988416,"internally_reserved":0,"allocated":2646016,"data_stored":148134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41877504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-11918,"num_objects_degraded":8117,"num_objects_misplaced":15413,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003629"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":4780,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:04.700665+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:04.700665+0000","last_peered":"2021-08-21T15:02:04.700665+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:04.700665+0000","last_undegraded":"2021-08-21T15:02:04.700665+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7296,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":275},{"shards":"2,3","objects":3648}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35038,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":261742592,"data_stored":411534,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964116996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247815,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672989,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21724,"kb_used_data":1300,"kb_used_omap":0,"kb_used_meta":20416,"kb_avail":94350116,"statfs":{"total":96636764160,"available":96614518784,"internally_reserved":0,"allocated":1331200,"data_stored":73910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20905984},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.58499999999999996}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.50700000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.54300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21772,"kb_used_data":1284,"kb_used_omap":0,"kb_used_meta":20480,"kb_avail":94350068,"statfs":{"total":96636764160,"available":96614469632,"internally_reserved":0,"allocated":1314816,"data_stored":74224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20971520},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.317},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.51400000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.56699999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1114112,"data_stored":272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:09.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:10.186 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:10.187 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:10.196 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2021-08-21T15:02:09.475605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35038,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":43764,"kb_used_data":2788,"kb_used_omap":0,"kb_used_meta":40960,"kb_avail":188699916,"statfs":{"total":193273528320,"available":193228713984,"internally_reserved":0,"allocated":2854912,"data_stored":148185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41943040},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-204,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003260"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":4780,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:04.700665+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:04.700665+0000","last_peered":"2021-08-21T15:02:04.700665+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:04.700665+0000","last_undegraded":"2021-08-21T15:02:04.700665+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7296,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":275},{"shards":"2,3","objects":3648}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35038,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":261951488,"data_stored":411585,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964116996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247815,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672989,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21724,"kb_used_data":1300,"kb_used_omap":0,"kb_used_meta":20416,"kb_avail":94350116,"statfs":{"total":96636764160,"available":96614518784,"internally_reserved":0,"allocated":1331200,"data_stored":73910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20905984},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.58499999999999996}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.50700000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.54300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705695,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22040,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":20544,"kb_avail":94349800,"statfs":{"total":96636764160,"available":96614195200,"internally_reserved":0,"allocated":1523712,"data_stored":74275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21037056},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.35299999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.504}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.67400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1282048,"data_stored":313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1114112,"data_stored":272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:10.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:10.503 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:10.504 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":130,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":44814336,"bytes_avail":193228713984,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":35038,"misplaced_total":63330,"misplaced_ratio":0.55326069793147004,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":102,"num_bytes_recovered":102,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2021-08-21T15:01:11.459247+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (75s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (75s)\n [............................] (remaining: 11h)","progress":0.0017681134631857276,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (75s)\n [............................] (remaining: 11h)","progress":0.0017681134631857276,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (70s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (75s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:10.515 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:10.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:10.777 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:10.780 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:10.789 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2021-08-21T15:02:09.475605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35038,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":43764,"kb_used_data":2788,"kb_used_omap":0,"kb_used_meta":40960,"kb_avail":188699916,"statfs":{"total":193273528320,"available":193228713984,"internally_reserved":0,"allocated":2854912,"data_stored":148185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41943040},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-204,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003260"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":4780,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:04.700665+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:04.700665+0000","last_peered":"2021-08-21T15:02:04.700665+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:04.700665+0000","last_undegraded":"2021-08-21T15:02:04.700665+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7296,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":275},{"shards":"2,3","objects":3648}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":35038,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":261951488,"data_stored":411585,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964116996,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247815,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672989,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":21724,"kb_used_data":1300,"kb_used_omap":0,"kb_used_meta":20416,"kb_avail":94350116,"statfs":{"total":96636764160,"available":96614518784,"internally_reserved":0,"allocated":1331200,"data_stored":73910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20905984},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.58499999999999996}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.50700000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.54300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705695,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22040,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":20544,"kb_avail":94349800,"statfs":{"total":96636764160,"available":96614195200,"internally_reserved":0,"allocated":1523712,"data_stored":74275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21037056},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.35299999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.504}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.67400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1282048,"data_stored":313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1114112,"data_stored":272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:13.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:14.060 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:14.062 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:14.071 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2021-08-21T15:02:13.476823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34936,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":44100,"kb_used_data":2996,"kb_used_omap":0,"kb_used_meta":41088,"kb_avail":188699580,"statfs":{"total":193273528320,"available":193228369920,"internally_reserved":0,"allocated":3067904,"data_stored":148237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42074112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-204,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003277"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":4933,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:09.653894+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:09.653894+0000","last_peered":"2021-08-21T15:02:09.653894+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:09.653894+0000","last_undegraded":"2021-08-21T15:02:09.653894+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7194,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":326},{"shards":"2,3","objects":3597}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34936,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262164480,"data_stored":411637,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964116997,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247816,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672990,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22060,"kb_used_data":1508,"kb_used_omap":0,"kb_used_meta":20544,"kb_avail":94349780,"statfs":{"total":96636764160,"available":96614174720,"internally_reserved":0,"allocated":1544192,"data_stored":73962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21037056},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.42599999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.41699999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.505},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.34999999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705695,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22040,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":20544,"kb_avail":94349800,"statfs":{"total":96636764160,"available":96614195200,"internally_reserved":0,"allocated":1523712,"data_stored":74275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21037056},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.35299999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.504}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.67400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1282048,"data_stored":313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1327104,"data_stored":324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:14.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:14.374 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:14.375 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":134,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":45158400,"bytes_avail":193228369920,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":34936,"misplaced_total":63330,"misplaced_ratio":0.55165008684667616,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":102,"num_bytes_recovered":102,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2021-08-21T15:01:11.459247+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (75s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (75s)\n [............................] (remaining: 11h)","progress":0.0017681134631857276,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (75s)\n [............................] (remaining: 11h)","progress":0.0017681134631857276,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (70s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (75s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:14.386 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:14.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:14.642 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:14.644 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:14.653 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2021-08-21T15:02:13.476823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34936,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":44100,"kb_used_data":2996,"kb_used_omap":0,"kb_used_meta":41088,"kb_avail":188699580,"statfs":{"total":193273528320,"available":193228369920,"internally_reserved":0,"allocated":3067904,"data_stored":148237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42074112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-204,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003277"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":4933,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:09.653894+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:09.653894+0000","last_peered":"2021-08-21T15:02:09.653894+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:09.653894+0000","last_undegraded":"2021-08-21T15:02:09.653894+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7194,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":326},{"shards":"2,3","objects":3597}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34936,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262164480,"data_stored":411637,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964116997,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247816,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672990,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22060,"kb_used_data":1508,"kb_used_omap":0,"kb_used_meta":20544,"kb_avail":94349780,"statfs":{"total":96636764160,"available":96614174720,"internally_reserved":0,"allocated":1544192,"data_stored":73962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21037056},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.42599999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.41699999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.505},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.34999999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705695,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22040,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":20544,"kb_avail":94349800,"statfs":{"total":96636764160,"available":96614195200,"internally_reserved":0,"allocated":1523712,"data_stored":74275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21037056},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.35299999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.504}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.67400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1282048,"data_stored":313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1327104,"data_stored":324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:17.655 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:17.918 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:17.921 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:17.930 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2021-08-21T15:02:17.478110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34832,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":44640,"kb_used_data":3408,"kb_used_omap":0,"kb_used_meta":41216,"kb_avail":188699040,"statfs":{"total":193273528320,"available":193227816960,"internally_reserved":0,"allocated":3489792,"data_stored":148340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42205184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003387"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5089,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:14.714817+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:14.714817+0000","last_peered":"2021-08-21T15:02:14.714817+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:14.714817+0000","last_undegraded":"2021-08-21T15:02:14.714817+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7090,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":378},{"shards":"2,3","objects":3545}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34832,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262586368,"data_stored":411740,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964116998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247817,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672991,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22328,"kb_used_data":1712,"kb_used_omap":0,"kb_used_meta":20608,"kb_avail":94349512,"statfs":{"total":96636764160,"available":96613900288,"internally_reserved":0,"allocated":1753088,"data_stored":74013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21102592},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.26900000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56200000000000006}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.50900000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22312,"kb_used_data":1696,"kb_used_omap":0,"kb_used_meta":20608,"kb_avail":94349528,"statfs":{"total":96636764160,"available":96613916672,"internally_reserved":0,"allocated":1736704,"data_stored":74327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21102592},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.251}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.621}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1495040,"data_stored":365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1536000,"data_stored":375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:17.931 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:18.235 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:18.235 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":138,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":45711360,"bytes_avail":193227816960,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":34832,"misplaced_total":63330,"misplaced_ratio":0.55000789515237647,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2021-08-21T15:01:11.459247+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (80s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (80s)\n [............................] (remaining: 10h)","progress":0.0020960182882845402,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (80s)\n [............................] (remaining: 10h)","progress":0.0020960182882845402,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (75s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (80s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:18.247 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:18.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:18.505 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:18.507 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:18.520 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2021-08-21T15:02:17.478110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34832,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":44640,"kb_used_data":3408,"kb_used_omap":0,"kb_used_meta":41216,"kb_avail":188699040,"statfs":{"total":193273528320,"available":193227816960,"internally_reserved":0,"allocated":3489792,"data_stored":148340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42205184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003387"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5089,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:14.714817+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:14.714817+0000","last_peered":"2021-08-21T15:02:14.714817+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:14.714817+0000","last_undegraded":"2021-08-21T15:02:14.714817+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7090,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":378},{"shards":"2,3","objects":3545}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34832,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262586368,"data_stored":411740,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964116998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247817,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672991,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22328,"kb_used_data":1712,"kb_used_omap":0,"kb_used_meta":20608,"kb_avail":94349512,"statfs":{"total":96636764160,"available":96613900288,"internally_reserved":0,"allocated":1753088,"data_stored":74013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21102592},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.26900000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56200000000000006}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.50900000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22312,"kb_used_data":1696,"kb_used_omap":0,"kb_used_meta":20608,"kb_avail":94349528,"statfs":{"total":96636764160,"available":96613916672,"internally_reserved":0,"allocated":1736704,"data_stored":74327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21102592},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.251}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.621}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1495040,"data_stored":365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1536000,"data_stored":375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:21.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:21.771 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:21.773 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:21.782 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2021-08-21T15:02:21.479230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34730,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":429,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":45180,"kb_used_data":3820,"kb_used_omap":0,"kb_used_meta":41344,"kb_avail":188698500,"statfs":{"total":193273528320,"available":193227264000,"internally_reserved":0,"allocated":3911680,"data_stored":148443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42336256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-308,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003624"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5242,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:19.679542+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:19.679542+0000","last_peered":"2021-08-21T15:02:19.679542+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:19.679542+0000","last_undegraded":"2021-08-21T15:02:19.679542+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6988,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":429,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":429},{"shards":"2,3","objects":3494}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34730,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":429,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":263008256,"data_stored":411843,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964116999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247818,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672992,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22600,"kb_used_data":1920,"kb_used_omap":0,"kb_used_meta":20672,"kb_avail":94349240,"statfs":{"total":96636764160,"available":96613621760,"internally_reserved":0,"allocated":1966080,"data_stored":74065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21168128},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.34200000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.39900000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705697,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22580,"kb_used_data":1900,"kb_used_omap":0,"kb_used_meta":20672,"kb_avail":94349260,"statfs":{"total":96636764160,"available":96613642240,"internally_reserved":0,"allocated":1945600,"data_stored":74378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21168128},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.439},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.45700000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.59399999999999997}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.53600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1703936,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1748992,"data_stored":427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:21.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:22.091 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:22.091 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":142,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":46264320,"bytes_avail":193227264000,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":34730,"misplaced_total":63330,"misplaced_ratio":0.54839728406758248,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":12,"recovering_keys_per_sec":0,"num_objects_recovered":154,"num_bytes_recovered":154,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2021-08-21T15:01:11.459247+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (85s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (85s)\n [............................] (remaining: 9h)","progress":0.0024303523823618889,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (85s)\n [............................] (remaining: 9h)","progress":0.0024303523823618889,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (80s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (85s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:22.103 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:22.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:22.351 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:22.353 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:22.367 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2021-08-21T15:02:21.479230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34730,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":429,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":45180,"kb_used_data":3820,"kb_used_omap":0,"kb_used_meta":41344,"kb_avail":188698500,"statfs":{"total":193273528320,"available":193227264000,"internally_reserved":0,"allocated":3911680,"data_stored":148443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42336256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-308,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003624"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5242,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:19.679542+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:19.679542+0000","last_peered":"2021-08-21T15:02:19.679542+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:19.679542+0000","last_undegraded":"2021-08-21T15:02:19.679542+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6988,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":429,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":429},{"shards":"2,3","objects":3494}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34730,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":429,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":263008256,"data_stored":411843,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964116999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247818,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672992,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22600,"kb_used_data":1920,"kb_used_omap":0,"kb_used_meta":20672,"kb_avail":94349240,"statfs":{"total":96636764160,"available":96613621760,"internally_reserved":0,"allocated":1966080,"data_stored":74065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21168128},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.34200000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.39900000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705697,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22580,"kb_used_data":1900,"kb_used_omap":0,"kb_used_meta":20672,"kb_avail":94349260,"statfs":{"total":96636764160,"available":96613642240,"internally_reserved":0,"allocated":1945600,"data_stored":74378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21168128},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.439},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.45700000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.59399999999999997}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.53600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1703936,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1748992,"data_stored":427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:25.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:25.638 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:25.640 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:25.656 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2021-08-21T15:02:25.480409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34626,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":45844,"kb_used_data":4228,"kb_used_omap":0,"kb_used_meta":41600,"kb_avail":188697836,"statfs":{"total":193273528320,"available":193226584064,"internally_reserved":0,"allocated":4329472,"data_stored":148545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42598400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-310,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003586"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5398,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:24.694700+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:24.694700+0000","last_peered":"2021-08-21T15:02:24.694700+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:24.694700+0000","last_undegraded":"2021-08-21T15:02:24.694700+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6884,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":481},{"shards":"2,3","objects":3442}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34626,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":263430144,"data_stored":411946,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117000,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247819,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672993,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22936,"kb_used_data":2128,"kb_used_omap":0,"kb_used_meta":20800,"kb_avail":94348904,"statfs":{"total":96636764160,"available":96613277696,"internally_reserved":0,"allocated":2179072,"data_stored":74117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21299200},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.621}]}]},{"osd":0,"up_from":9,"seq":38654705698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22908,"kb_used_data":2100,"kb_used_omap":0,"kb_used_meta":20800,"kb_avail":94348932,"statfs":{"total":96636764160,"available":96613306368,"internally_reserved":0,"allocated":2150400,"data_stored":74428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21299200},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.434},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.58799999999999997}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1916928,"data_stored":468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1957888,"data_stored":478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:25.657 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:25.968 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:25.968 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":146,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":46264320,"bytes_avail":193227264000,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":34730,"misplaced_total":63330,"misplaced_ratio":0.54839728406758248,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2021-08-21T15:01:11.459247+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (85s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (85s)\n [............................] (remaining: 9h)","progress":0.0024303523823618889,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (85s)\n [............................] (remaining: 9h)","progress":0.0024303523823618889,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (80s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (85s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:25.982 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:25.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:26.248 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:26.249 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:26.259 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2021-08-21T15:02:25.480409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34626,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":45844,"kb_used_data":4228,"kb_used_omap":0,"kb_used_meta":41600,"kb_avail":188697836,"statfs":{"total":193273528320,"available":193226584064,"internally_reserved":0,"allocated":4329472,"data_stored":148545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42598400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-310,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003586"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5398,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:24.694700+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:24.694700+0000","last_peered":"2021-08-21T15:02:24.694700+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:24.694700+0000","last_undegraded":"2021-08-21T15:02:24.694700+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6884,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":481},{"shards":"2,3","objects":3442}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34626,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":263430144,"data_stored":411946,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117000,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247819,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672993,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22936,"kb_used_data":2128,"kb_used_omap":0,"kb_used_meta":20800,"kb_avail":94348904,"statfs":{"total":96636764160,"available":96613277696,"internally_reserved":0,"allocated":2179072,"data_stored":74117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21299200},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.621}]}]},{"osd":0,"up_from":9,"seq":38654705698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22908,"kb_used_data":2100,"kb_used_omap":0,"kb_used_meta":20800,"kb_avail":94348932,"statfs":{"total":96636764160,"available":96613306368,"internally_reserved":0,"allocated":2150400,"data_stored":74428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21299200},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.45500000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.434},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.58799999999999997}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1916928,"data_stored":468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1957888,"data_stored":478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:29.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:29.519 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:29.522 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:29.533 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2021-08-21T15:02:29.481601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34626,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":46112,"kb_used_data":4432,"kb_used_omap":0,"kb_used_meta":41664,"kb_avail":188697568,"statfs":{"total":193273528320,"available":193226309632,"internally_reserved":0,"allocated":4538368,"data_stored":148596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42663936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003490"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5398,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:24.694700+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:24.694700+0000","last_peered":"2021-08-21T15:02:24.694700+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:24.694700+0000","last_undegraded":"2021-08-21T15:02:24.694700+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6884,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":481},{"shards":"2,3","objects":3442}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34626,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":263643136,"data_stored":411998,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117000,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247819,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672993,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22936,"kb_used_data":2128,"kb_used_omap":0,"kb_used_meta":20800,"kb_avail":94348904,"statfs":{"total":96636764160,"available":96613277696,"internally_reserved":0,"allocated":2179072,"data_stored":74117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21299200},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.621}]}]},{"osd":0,"up_from":9,"seq":38654705699,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23176,"kb_used_data":2304,"kb_used_omap":0,"kb_used_meta":20864,"kb_avail":94348664,"statfs":{"total":96636764160,"available":96613031936,"internally_reserved":0,"allocated":2359296,"data_stored":74479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21364736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.442},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.309}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.63200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1957888,"data_stored":478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:29.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:29.838 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:29.839 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":150,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":46944256,"bytes_avail":193226584064,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":34626,"misplaced_total":63330,"misplaced_ratio":0.54675509237328279,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2021-08-21T15:01:11.459247+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (90s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (90s)\n [............................] (remaining: 9h)","progress":0.0027582570910453796,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (90s)\n [............................] (remaining: 9h)","progress":0.0027582570910453796,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (85s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (90s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:29.854 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:29.855 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:30.133 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:30.143 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2021-08-21T15:02:29.481601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34626,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":46112,"kb_used_data":4432,"kb_used_omap":0,"kb_used_meta":41664,"kb_avail":188697568,"statfs":{"total":193273528320,"available":193226309632,"internally_reserved":0,"allocated":4538368,"data_stored":148596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42663936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003490"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5398,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:24.694700+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:24.694700+0000","last_peered":"2021-08-21T15:02:24.694700+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:24.694700+0000","last_undegraded":"2021-08-21T15:02:24.694700+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6884,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":481},{"shards":"2,3","objects":3442}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34626,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":263643136,"data_stored":411998,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117000,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247819,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672993,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":22936,"kb_used_data":2128,"kb_used_omap":0,"kb_used_meta":20800,"kb_avail":94348904,"statfs":{"total":96636764160,"available":96613277696,"internally_reserved":0,"allocated":2179072,"data_stored":74117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21299200},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.621}]}]},{"osd":0,"up_from":9,"seq":38654705699,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23176,"kb_used_data":2304,"kb_used_omap":0,"kb_used_meta":20864,"kb_avail":94348664,"statfs":{"total":96636764160,"available":96613031936,"internally_reserved":0,"allocated":2359296,"data_stored":74479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21364736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.442},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.309}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.63200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1957888,"data_stored":478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:33.145 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:33.408 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:33.409 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:33.416 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2021-08-21T15:02:31.482157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34522,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":533,"num_bytes_recovered":533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":46384,"kb_used_data":4640,"kb_used_omap":0,"kb_used_meta":41728,"kb_avail":188697296,"statfs":{"total":193273528320,"available":193226031104,"internally_reserved":0,"allocated":4751360,"data_stored":148648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42729472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-310,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003492"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5554,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:29.731638+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:29.731638+0000","last_peered":"2021-08-21T15:02:29.731638+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:29.731638+0000","last_undegraded":"2021-08-21T15:02:29.731638+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6780,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":533,"num_bytes_recovered":533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":533},{"shards":"2,3","objects":3390}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34522,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":533,"num_bytes_recovered":533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":263856128,"data_stored":412050,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117001,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672994,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23208,"kb_used_data":2336,"kb_used_omap":0,"kb_used_meta":20864,"kb_avail":94348632,"statfs":{"total":96636764160,"available":96612999168,"internally_reserved":0,"allocated":2392064,"data_stored":74169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21364736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.621}]}]},{"osd":0,"up_from":9,"seq":38654705699,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23176,"kb_used_data":2304,"kb_used_omap":0,"kb_used_meta":20864,"kb_avail":94348664,"statfs":{"total":96636764160,"available":96613031936,"internally_reserved":0,"allocated":2359296,"data_stored":74479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21364736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.442},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.309}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.63200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2170880,"data_stored":530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:33.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:33.780 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:33.780 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":154,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":47497216,"bytes_avail":193226031104,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":34522,"misplaced_total":63330,"misplaced_ratio":0.5451129006789831,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":12,"recovering_keys_per_sec":0,"num_objects_recovered":155,"num_bytes_recovered":155,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2021-08-21T15:01:11.459247+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (95s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (95s)\n [............................] (remaining: 8h)","progress":0.0030925911851227283,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (95s)\n [............................] (remaining: 8h)","progress":0.0030925911851227283,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (90s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (95s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:33.787 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:33.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:34.051 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:34.053 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:34.061 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2021-08-21T15:02:33.482727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34522,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":533,"num_bytes_recovered":533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":46384,"kb_used_data":4640,"kb_used_omap":0,"kb_used_meta":41728,"kb_avail":188697296,"statfs":{"total":193273528320,"available":193226031104,"internally_reserved":0,"allocated":4751360,"data_stored":148648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42729472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003496"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5554,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:29.731638+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:29.731638+0000","last_peered":"2021-08-21T15:02:29.731638+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:29.731638+0000","last_undegraded":"2021-08-21T15:02:29.731638+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6780,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":533,"num_bytes_recovered":533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":533},{"shards":"2,3","objects":3390}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34522,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":533,"num_bytes_recovered":533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":263856128,"data_stored":412050,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117001,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672994,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23208,"kb_used_data":2336,"kb_used_omap":0,"kb_used_meta":20864,"kb_avail":94348632,"statfs":{"total":96636764160,"available":96612999168,"internally_reserved":0,"allocated":2392064,"data_stored":74169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21364736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52700000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.621}]}]},{"osd":0,"up_from":9,"seq":38654705699,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23176,"kb_used_data":2304,"kb_used_omap":0,"kb_used_meta":20864,"kb_avail":94348664,"statfs":{"total":96636764160,"available":96613031936,"internally_reserved":0,"allocated":2359296,"data_stored":74479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21364736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.442},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.309}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.63200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2170880,"data_stored":530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:37.065 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:37.325 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:37.327 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:37.336 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2021-08-21T15:02:35.483179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34420,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":46924,"kb_used_data":5052,"kb_used_omap":0,"kb_used_meta":41856,"kb_avail":188696756,"statfs":{"total":193273528320,"available":193225478144,"internally_reserved":0,"allocated":5173248,"data_stored":148751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42860544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-310,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003312"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5707,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:34.677755+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:34.677755+0000","last_peered":"2021-08-21T15:02:34.677755+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:34.677755+0000","last_undegraded":"2021-08-21T15:02:34.677755+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6678,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":584},{"shards":"2,3","objects":3339}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34420,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":264278016,"data_stored":412153,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117002,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247821,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672995,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23472,"kb_used_data":2536,"kb_used_omap":0,"kb_used_meta":20928,"kb_avail":94348368,"statfs":{"total":96636764160,"available":96612728832,"internally_reserved":0,"allocated":2596864,"data_stored":74219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21430272},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.438}]}]},{"osd":0,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23452,"kb_used_data":2516,"kb_used_omap":0,"kb_used_meta":20928,"kb_avail":94348388,"statfs":{"total":96636764160,"available":96612749312,"internally_reserved":0,"allocated":2576384,"data_stored":74532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21430272},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.51400000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.44800000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.50800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2338816,"data_stored":571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2383872,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:37.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:37.656 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:37.656 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":157,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":48050176,"bytes_avail":193225478144,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":34420,"misplaced_total":63330,"misplaced_ratio":0.54350228959418911,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":12,"recovering_keys_per_sec":0,"num_objects_recovered":155,"num_bytes_recovered":155,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2021-08-21T15:02:35.483282+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (100s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (100s)\n [............................] (remaining: 8h)","progress":0.0034269255120307207,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (100s)\n [............................] (remaining: 8h)","progress":0.0034269255120307207,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (95s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (100s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:37.667 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:37.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:37.916 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:37.918 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:37.930 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2021-08-21T15:02:37.483805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34420,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":46924,"kb_used_data":5052,"kb_used_omap":0,"kb_used_meta":41856,"kb_avail":188696756,"statfs":{"total":193273528320,"available":193225478144,"internally_reserved":0,"allocated":5173248,"data_stored":148751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42860544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003395"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5707,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:34.677755+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:34.677755+0000","last_peered":"2021-08-21T15:02:34.677755+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:34.677755+0000","last_undegraded":"2021-08-21T15:02:34.677755+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6678,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":584},{"shards":"2,3","objects":3339}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34420,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":264278016,"data_stored":412153,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117002,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247821,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672995,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23472,"kb_used_data":2536,"kb_used_omap":0,"kb_used_meta":20928,"kb_avail":94348368,"statfs":{"total":96636764160,"available":96612728832,"internally_reserved":0,"allocated":2596864,"data_stored":74219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21430272},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.438}]}]},{"osd":0,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23452,"kb_used_data":2516,"kb_used_omap":0,"kb_used_meta":20928,"kb_avail":94348388,"statfs":{"total":96636764160,"available":96612749312,"internally_reserved":0,"allocated":2576384,"data_stored":74532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21430272},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.51400000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.44800000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.50800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2338816,"data_stored":571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2383872,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:40.940 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:41.205 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:41.209 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:41.219 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2021-08-21T15:02:39.484529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34420,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":51356,"kb_used_data":5260,"kb_used_omap":0,"kb_used_meta":46080,"kb_avail":188692324,"statfs":{"total":193273528320,"available":193220939776,"internally_reserved":0,"allocated":5386240,"data_stored":148803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47185920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003477"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5707,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:34.677755+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:34.677755+0000","last_peered":"2021-08-21T15:02:34.677755+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:34.677755+0000","last_undegraded":"2021-08-21T15:02:34.677755+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6678,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":584},{"shards":"2,3","objects":3339}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34420,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":264491008,"data_stored":412205,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117002,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247821,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672995,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":23472,"kb_used_data":2536,"kb_used_omap":0,"kb_used_meta":20928,"kb_avail":94348368,"statfs":{"total":96636764160,"available":96612728832,"internally_reserved":0,"allocated":2596864,"data_stored":74219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21430272},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.438}]}]},{"osd":0,"up_from":9,"seq":38654705701,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":2724,"kb_used_omap":0,"kb_used_meta":25152,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":2789376,"data_stored":74584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25755648},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.34999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.373},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.45100000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.496}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2551808,"data_stored":623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2383872,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:41.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:41.530 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:41.530 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 5 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":161,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":52588544,"bytes_avail":193220939776,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":34420,"misplaced_total":63330,"misplaced_ratio":0.54350228959418911,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2021-08-21T15:02:35.483282+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (105s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (105s)\n [............................] (remaining: 7h)","progress":0.0037548302207142115,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (105s)\n [............................] (remaining: 7h)","progress":0.0037548302207142115,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (100s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (105s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:41.544 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:41.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:41.823 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:41.825 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:41.836 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2021-08-21T15:02:41.484969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34316,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":636,"num_bytes_recovered":636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":55788,"kb_used_data":5468,"kb_used_omap":0,"kb_used_meta":50304,"kb_avail":188687892,"statfs":{"total":193273528320,"available":193216401408,"internally_reserved":0,"allocated":5599232,"data_stored":148855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51511296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-310,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003368"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5863,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:39.716938+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:39.716938+0000","last_peered":"2021-08-21T15:02:39.716938+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:39.716938+0000","last_undegraded":"2021-08-21T15:02:39.716938+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6574,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":636,"num_bytes_recovered":636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":636},{"shards":"2,3","objects":3287}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34316,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":636,"num_bytes_recovered":636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":264699904,"data_stored":412256,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117003,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247822,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672996,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":2744,"kb_used_omap":0,"kb_used_meta":25152,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":2809856,"data_stored":74271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25755648},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.438}]}]},{"osd":0,"up_from":9,"seq":38654705701,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":2724,"kb_used_omap":0,"kb_used_meta":25152,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":2789376,"data_stored":74584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25755648},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.34999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.373},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.45100000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.496}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2551808,"data_stored":623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2592768,"data_stored":633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:44.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:45.109 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:45.111 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:45.123 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2021-08-21T15:02:43.485602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34316,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":636,"num_bytes_recovered":636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":55788,"kb_used_data":5468,"kb_used_omap":0,"kb_used_meta":50304,"kb_avail":188687892,"statfs":{"total":193273528320,"available":193216401408,"internally_reserved":0,"allocated":5599232,"data_stored":148855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51511296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003444"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":5863,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:39.716938+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:39.716938+0000","last_peered":"2021-08-21T15:02:39.716938+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:39.716938+0000","last_undegraded":"2021-08-21T15:02:39.716938+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6574,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":636,"num_bytes_recovered":636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":636},{"shards":"2,3","objects":3287}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34316,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":636,"num_bytes_recovered":636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":264699904,"data_stored":412256,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117003,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247822,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672996,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27904,"kb_used_data":2744,"kb_used_omap":0,"kb_used_meta":25152,"kb_avail":94343936,"statfs":{"total":96636764160,"available":96608190464,"internally_reserved":0,"allocated":2809856,"data_stored":74271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25755648},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.438}]}]},{"osd":0,"up_from":9,"seq":38654705701,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27884,"kb_used_data":2724,"kb_used_omap":0,"kb_used_meta":25152,"kb_avail":94343956,"statfs":{"total":96636764160,"available":96608210944,"internally_reserved":0,"allocated":2789376,"data_stored":74584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25755648},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.34999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.373},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.45100000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.496}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2551808,"data_stored":623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2592768,"data_stored":633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:45.124 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:45.419 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:45.419 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 5 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":165,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":57126912,"bytes_avail":193216401408,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":34316,"misplaced_total":63330,"misplaced_ratio":0.54186009789988943,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2021-08-21T15:02:35.483282+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (105s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (105s)\n [............................] (remaining: 7h)","progress":0.0037548302207142115,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (105s)\n [............................] (remaining: 7h)","progress":0.0037548302207142115,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (100s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (105s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:45.431 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:45.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:45.689 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:45.691 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:45.701 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2021-08-21T15:02:45.486398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34212,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":56332,"kb_used_data":5884,"kb_used_omap":0,"kb_used_meta":50432,"kb_avail":188687348,"statfs":{"total":193273528320,"available":193215844352,"internally_reserved":0,"allocated":6025216,"data_stored":148959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51642368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-310,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003671"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6019,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:44.739759+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:44.739759+0000","last_peered":"2021-08-21T15:02:44.739759+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:44.739759+0000","last_undegraded":"2021-08-21T15:02:44.739759+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6470,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":688},{"shards":"2,3","objects":3235}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34212,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":265125888,"data_stored":412360,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247823,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672997,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28176,"kb_used_data":2952,"kb_used_omap":0,"kb_used_meta":25216,"kb_avail":94343664,"statfs":{"total":96636764160,"available":96607911936,"internally_reserved":0,"allocated":3022848,"data_stored":74323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25821184},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.39700000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.60999999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705702,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":2932,"kb_used_omap":0,"kb_used_meta":25216,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":3002368,"data_stored":74636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25821184},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.252},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.38}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.46300000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2764800,"data_stored":675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2805760,"data_stored":685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:48.702 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:48.966 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:48.968 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:48.980 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2021-08-21T15:02:47.487068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34212,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":56332,"kb_used_data":5884,"kb_used_omap":0,"kb_used_meta":50432,"kb_avail":188687348,"statfs":{"total":193273528320,"available":193215844352,"internally_reserved":0,"allocated":6025216,"data_stored":148959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51642368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003888"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6019,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:44.739759+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:44.739759+0000","last_peered":"2021-08-21T15:02:44.739759+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:44.739759+0000","last_undegraded":"2021-08-21T15:02:44.739759+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6470,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":688},{"shards":"2,3","objects":3235}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34212,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":265125888,"data_stored":412360,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247823,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672997,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28176,"kb_used_data":2952,"kb_used_omap":0,"kb_used_meta":25216,"kb_avail":94343664,"statfs":{"total":96636764160,"available":96607911936,"internally_reserved":0,"allocated":3022848,"data_stored":74323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25821184},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.39700000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.60999999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705702,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":2932,"kb_used_omap":0,"kb_used_meta":25216,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":3002368,"data_stored":74636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25821184},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.252},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.38}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.46300000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.50900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2764800,"data_stored":675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2805760,"data_stored":685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:48.982 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:49.289 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:49.290 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 5 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":169,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":57683968,"bytes_avail":193215844352,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":34212,"misplaced_total":63330,"misplaced_ratio":0.54021790620558974,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2021-08-21T15:02:35.483282+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (110s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (110s)\n [............................] (remaining: 7h)","progress":0.0040891645476222038,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (110s)\n [............................] (remaining: 7h)","progress":0.0040891645476222038,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (105s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (110s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:49.301 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:49.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:49.565 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:49.568 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:49.578 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2021-08-21T15:02:49.487618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34212,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":56608,"kb_used_data":6096,"kb_used_omap":0,"kb_used_meta":50496,"kb_avail":188687072,"statfs":{"total":193273528320,"available":193215561728,"internally_reserved":0,"allocated":6242304,"data_stored":149012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51707904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003813"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6019,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:44.739759+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:44.739759+0000","last_peered":"2021-08-21T15:02:44.739759+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:44.739759+0000","last_undegraded":"2021-08-21T15:02:44.739759+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6470,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":688},{"shards":"2,3","objects":3235}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34212,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":265338880,"data_stored":412412,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247823,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672997,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28176,"kb_used_data":2952,"kb_used_omap":0,"kb_used_meta":25216,"kb_avail":94343664,"statfs":{"total":96636764160,"available":96607911936,"internally_reserved":0,"allocated":3022848,"data_stored":74323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25821184},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.39700000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.53600000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.60999999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28432,"kb_used_data":3144,"kb_used_omap":0,"kb_used_meta":25280,"kb_avail":94343408,"statfs":{"total":96636764160,"available":96607649792,"internally_reserved":0,"allocated":3219456,"data_stored":74689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25886720},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.437},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.57699999999999996}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.621},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2977792,"data_stored":727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2805760,"data_stored":685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:52.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:52.845 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:52.852 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2021-08-21T15:02:51.488161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34110,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":739,"num_bytes_recovered":739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":56884,"kb_used_data":6308,"kb_used_omap":0,"kb_used_meta":50560,"kb_avail":188686796,"statfs":{"total":193273528320,"available":193215279104,"internally_reserved":0,"allocated":6459392,"data_stored":149065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51773440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-310,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003631"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6172,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:49.651288+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:49.651288+0000","last_peered":"2021-08-21T15:02:49.651288+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:49.651288+0000","last_undegraded":"2021-08-21T15:02:49.651288+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6368,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":739,"num_bytes_recovered":739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":739},{"shards":"2,3","objects":3184}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34110,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":739,"num_bytes_recovered":739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":265551872,"data_stored":412464,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117005,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672998,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28452,"kb_used_data":3164,"kb_used_omap":0,"kb_used_meta":25280,"kb_avail":94343388,"statfs":{"total":96636764160,"available":96607629312,"internally_reserved":0,"allocated":3239936,"data_stored":74376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25886720},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.372}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.57199999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28432,"kb_used_data":3144,"kb_used_omap":0,"kb_used_meta":25280,"kb_avail":94343408,"statfs":{"total":96636764160,"available":96607649792,"internally_reserved":0,"allocated":3219456,"data_stored":74689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25886720},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.437},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.57699999999999996}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.621},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2977792,"data_stored":727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3018752,"data_stored":737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:52.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:53.169 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:53.170 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 5 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":173,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":58249216,"bytes_avail":193215279104,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":34110,"misplaced_total":63330,"misplaced_ratio":0.53860729512079586,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":12,"recovering_keys_per_sec":0,"num_objects_recovered":155,"num_bytes_recovered":155,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (115s)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (115s)\n [............................] (remaining: 7h)","progress":0.0044234986416995525,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (115s)\n [............................] (remaining: 7h)","progress":0.0044234986416995525,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (110s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (115s)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:53.183 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:53.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:53.461 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:53.464 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:53.473 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2021-08-21T15:02:51.488161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34110,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":739,"num_bytes_recovered":739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":56884,"kb_used_data":6308,"kb_used_omap":0,"kb_used_meta":50560,"kb_avail":188686796,"statfs":{"total":193273528320,"available":193215279104,"internally_reserved":0,"allocated":6459392,"data_stored":149065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51773440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-310,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003631"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6172,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:49.651288+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:49.651288+0000","last_peered":"2021-08-21T15:02:49.651288+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:49.651288+0000","last_undegraded":"2021-08-21T15:02:49.651288+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6368,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":739,"num_bytes_recovered":739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":739},{"shards":"2,3","objects":3184}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34110,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":739,"num_bytes_recovered":739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":265551872,"data_stored":412464,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117005,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672998,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28452,"kb_used_data":3164,"kb_used_omap":0,"kb_used_meta":25280,"kb_avail":94343388,"statfs":{"total":96636764160,"available":96607629312,"internally_reserved":0,"allocated":3239936,"data_stored":74376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25886720},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.372}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.49199999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:01:09 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.57199999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28432,"kb_used_data":3144,"kb_used_omap":0,"kb_used_meta":25280,"kb_avail":94343408,"statfs":{"total":96636764160,"available":96607649792,"internally_reserved":0,"allocated":3219456,"data_stored":74689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25886720},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.437},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.54500000000000004}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.57699999999999996}]},{"osd":3,"last update":"Sat Aug 21 15:01:10 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.621},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.42099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2977792,"data_stored":727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3018752,"data_stored":737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:56.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:56.755 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:56.757 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:56.766 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2021-08-21T15:02:55.489548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34006,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":791,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":57552,"kb_used_data":6720,"kb_used_omap":0,"kb_used_meta":50816,"kb_avail":188686128,"statfs":{"total":193273528320,"available":193214595072,"internally_reserved":0,"allocated":6881280,"data_stored":149168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52035584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-310,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003946"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6328,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:54.668590+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:54.668590+0000","last_peered":"2021-08-21T15:02:54.668590+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:54.668590+0000","last_undegraded":"2021-08-21T15:02:54.668590+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6264,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":791,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":791},{"shards":"2,3","objects":3132}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34006,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":791,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":265973760,"data_stored":412567,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117006,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247825,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672999,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28788,"kb_used_data":3372,"kb_used_omap":0,"kb_used_meta":25408,"kb_avail":94343052,"statfs":{"total":96636764160,"available":96607285248,"internally_reserved":0,"allocated":3452928,"data_stored":74428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26017792},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.442},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.27200000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.45900000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28764,"kb_used_data":3348,"kb_used_omap":0,"kb_used_meta":25408,"kb_avail":94343076,"statfs":{"total":96636764160,"available":96607309824,"internally_reserved":0,"allocated":3428352,"data_stored":74740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26017792},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.29899999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53100000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3186688,"data_stored":778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3231744,"data_stored":789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:02:56.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:02:57.107 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:57.107 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":177,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":58933248,"bytes_avail":193214595072,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":34006,"misplaced_total":63330,"misplaced_ratio":0.53696510342649617,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":12,"recovering_keys_per_sec":0,"num_objects_recovered":155,"num_bytes_recovered":155,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0047514033503830433,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0047514033503830433,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (115s)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:02:57.121 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:02:57.122 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:02:57.395 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:02:57.398 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:02:57.408 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2021-08-21T15:02:55.489548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34006,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":791,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":57552,"kb_used_data":6720,"kb_used_omap":0,"kb_used_meta":50816,"kb_avail":188686128,"statfs":{"total":193273528320,"available":193214595072,"internally_reserved":0,"allocated":6881280,"data_stored":149168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52035584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-310,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003946"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6328,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:54.668590+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:54.668590+0000","last_peered":"2021-08-21T15:02:54.668590+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:54.668590+0000","last_undegraded":"2021-08-21T15:02:54.668590+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6264,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":791,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":791},{"shards":"2,3","objects":3132}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34006,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":791,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":265973760,"data_stored":412567,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117006,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247825,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672999,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28788,"kb_used_data":3372,"kb_used_omap":0,"kb_used_meta":25408,"kb_avail":94343052,"statfs":{"total":96636764160,"available":96607285248,"internally_reserved":0,"allocated":3452928,"data_stored":74428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26017792},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.442},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.27200000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.45900000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28764,"kb_used_data":3348,"kb_used_omap":0,"kb_used_meta":25408,"kb_avail":94343076,"statfs":{"total":96636764160,"available":96607309824,"internally_reserved":0,"allocated":3428352,"data_stored":74740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26017792},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.29899999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53100000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3186688,"data_stored":778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3231744,"data_stored":789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:00.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:00.664 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:00.666 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:00.675 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2021-08-21T15:02:59.490823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34006,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":791,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":57824,"kb_used_data":6928,"kb_used_omap":0,"kb_used_meta":50880,"kb_avail":188685856,"statfs":{"total":193273528320,"available":193214316544,"internally_reserved":0,"allocated":7094272,"data_stored":149220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52101120},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003755"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6328,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:54.668590+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:54.668590+0000","last_peered":"2021-08-21T15:02:54.668590+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:54.668590+0000","last_undegraded":"2021-08-21T15:02:54.668590+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6264,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":791,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":791},{"shards":"2,3","objects":3132}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34006,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":791,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266186752,"data_stored":412619,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117006,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247825,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672999,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28788,"kb_used_data":3372,"kb_used_omap":0,"kb_used_meta":25408,"kb_avail":94343052,"statfs":{"total":96636764160,"available":96607285248,"internally_reserved":0,"allocated":3452928,"data_stored":74428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26017792},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.442},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.27200000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.45900000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29036,"kb_used_data":3556,"kb_used_omap":0,"kb_used_meta":25472,"kb_avail":94342804,"statfs":{"total":96636764160,"available":96607031296,"internally_reserved":0,"allocated":3641344,"data_stored":74792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26083328},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.374}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3399680,"data_stored":830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3231744,"data_stored":789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:00.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:00.986 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:00.986 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":181,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":59211776,"bytes_avail":193214316544,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":34006,"misplaced_total":63330,"misplaced_ratio":0.53696510342649617,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 6h)","progress":0.005085737444460392,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 6h)","progress":0.005085737444460392,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:01.003 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:01.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:01.249 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:01.263 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:01.264 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2021-08-21T15:02:59.490823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34006,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":791,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":57824,"kb_used_data":6928,"kb_used_omap":0,"kb_used_meta":50880,"kb_avail":188685856,"statfs":{"total":193273528320,"available":193214316544,"internally_reserved":0,"allocated":7094272,"data_stored":149220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52101120},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003755"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6328,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:54.668590+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:54.668590+0000","last_peered":"2021-08-21T15:02:54.668590+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:54.668590+0000","last_undegraded":"2021-08-21T15:02:54.668590+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6264,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":791,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":791},{"shards":"2,3","objects":3132}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":34006,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":791,"num_bytes_recovered":791,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266186752,"data_stored":412619,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117006,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247825,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949672999,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28788,"kb_used_data":3372,"kb_used_omap":0,"kb_used_meta":25408,"kb_avail":94343052,"statfs":{"total":96636764160,"available":96607285248,"internally_reserved":0,"allocated":3452928,"data_stored":74428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26017792},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.442},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.27200000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.45900000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29036,"kb_used_data":3556,"kb_used_omap":0,"kb_used_meta":25472,"kb_avail":94342804,"statfs":{"total":96636764160,"available":96607031296,"internally_reserved":0,"allocated":3641344,"data_stored":74792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26083328},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.374}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3399680,"data_stored":830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3231744,"data_stored":789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:04.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:04.555 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:04.556 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:04.564 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2021-08-21T15:03:03.492136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33902,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":58360,"kb_used_data":7336,"kb_used_omap":0,"kb_used_meta":51008,"kb_avail":188685320,"statfs":{"total":193273528320,"available":193213767680,"internally_reserved":0,"allocated":7512064,"data_stored":149322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52232192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003974"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6484,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:59.662041+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:59.662041+0000","last_peered":"2021-08-21T15:02:59.662041+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:59.662041+0000","last_undegraded":"2021-08-21T15:02:59.662041+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6160,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":843},{"shards":"2,3","objects":3080}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33902,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266612736,"data_stored":412723,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117007,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247826,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673000,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29056,"kb_used_data":3576,"kb_used_omap":0,"kb_used_meta":25472,"kb_avail":94342784,"statfs":{"total":96636764160,"available":96607010816,"internally_reserved":0,"allocated":3661824,"data_stored":74479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26083328},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.48499999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.501}]}]},{"osd":0,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29304,"kb_used_data":3760,"kb_used_omap":0,"kb_used_meta":25536,"kb_avail":94342536,"statfs":{"total":96636764160,"available":96606756864,"internally_reserved":0,"allocated":3850240,"data_stored":74843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26148864},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.45800000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.372},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.62}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.55800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3612672,"data_stored":882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3444736,"data_stored":841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:04.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:04.884 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:04.885 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":185,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":59760640,"bytes_avail":193213767680,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":33902,"misplaced_total":63330,"misplaced_ratio":0.53532291173219648,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 6h)","progress":0.005085737444460392,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 6h)","progress":0.005085737444460392,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:04.893 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:04.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:05.151 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:05.152 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:05.154 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2021-08-21T15:03:03.492136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33902,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":58360,"kb_used_data":7336,"kb_used_omap":0,"kb_used_meta":51008,"kb_avail":188685320,"statfs":{"total":193273528320,"available":193213767680,"internally_reserved":0,"allocated":7512064,"data_stored":149322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52232192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003974"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6484,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:02:59.662041+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:02:59.662041+0000","last_peered":"2021-08-21T15:02:59.662041+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:02:59.662041+0000","last_undegraded":"2021-08-21T15:02:59.662041+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6160,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":843},{"shards":"2,3","objects":3080}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33902,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":843,"num_bytes_recovered":843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266612736,"data_stored":412723,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117007,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247826,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673000,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29056,"kb_used_data":3576,"kb_used_omap":0,"kb_used_meta":25472,"kb_avail":94342784,"statfs":{"total":96636764160,"available":96607010816,"internally_reserved":0,"allocated":3661824,"data_stored":74479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26083328},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.48499999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.501}]}]},{"osd":0,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29304,"kb_used_data":3760,"kb_used_omap":0,"kb_used_meta":25536,"kb_avail":94342536,"statfs":{"total":96636764160,"available":96606756864,"internally_reserved":0,"allocated":3850240,"data_stored":74843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26148864},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.45800000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.372},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.62}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.55800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3612672,"data_stored":882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3444736,"data_stored":841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:08.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:08.451 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:08.453 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:08.454 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2021-08-21T15:03:07.493606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33798,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":895,"num_bytes_recovered":895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":58636,"kb_used_data":7548,"kb_used_omap":0,"kb_used_meta":51072,"kb_avail":188685044,"statfs":{"total":193273528320,"available":193213485056,"internally_reserved":0,"allocated":7729152,"data_stored":149375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52297728},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004058"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6640,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:04.685322+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:04.685322+0000","last_peered":"2021-08-21T15:03:04.685322+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:04.685322+0000","last_undegraded":"2021-08-21T15:03:04.685322+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6056,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":895,"num_bytes_recovered":895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":895},{"shards":"2,3","objects":3028}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33798,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":895,"num_bytes_recovered":895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266825728,"data_stored":412775,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117008,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247827,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673001,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29332,"kb_used_data":3788,"kb_used_omap":0,"kb_used_meta":25536,"kb_avail":94342508,"statfs":{"total":96636764160,"available":96606728192,"internally_reserved":0,"allocated":3878912,"data_stored":74532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26148864},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.312}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.28999999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.378},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.39400000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29304,"kb_used_data":3760,"kb_used_omap":0,"kb_used_meta":25536,"kb_avail":94342536,"statfs":{"total":96636764160,"available":96606756864,"internally_reserved":0,"allocated":3850240,"data_stored":74843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26148864},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.45800000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.372},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.62}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.55800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3612672,"data_stored":882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3657728,"data_stored":893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:08.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:08.759 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:08.759 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":189,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":60043264,"bytes_avail":193213485056,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":33798,"misplaced_total":63330,"misplaced_ratio":0.53368072003789668,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0054200715385377407,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0054200715385377407,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:08.771 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:08.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:09.035 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:09.037 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:09.047 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2021-08-21T15:03:07.493606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33798,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":895,"num_bytes_recovered":895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":58636,"kb_used_data":7548,"kb_used_omap":0,"kb_used_meta":51072,"kb_avail":188685044,"statfs":{"total":193273528320,"available":193213485056,"internally_reserved":0,"allocated":7729152,"data_stored":149375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52297728},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004058"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6640,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:04.685322+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:04.685322+0000","last_peered":"2021-08-21T15:03:04.685322+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:04.685322+0000","last_undegraded":"2021-08-21T15:03:04.685322+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6056,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":895,"num_bytes_recovered":895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":895},{"shards":"2,3","objects":3028}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33798,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":895,"num_bytes_recovered":895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266825728,"data_stored":412775,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117008,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247827,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673001,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29332,"kb_used_data":3788,"kb_used_omap":0,"kb_used_meta":25536,"kb_avail":94342508,"statfs":{"total":96636764160,"available":96606728192,"internally_reserved":0,"allocated":3878912,"data_stored":74532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26148864},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.312}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.28999999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.378},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.39400000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29304,"kb_used_data":3760,"kb_used_omap":0,"kb_used_meta":25536,"kb_avail":94342536,"statfs":{"total":96636764160,"available":96606756864,"internally_reserved":0,"allocated":3850240,"data_stored":74843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26148864},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.45800000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.372},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.62}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.55800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3612672,"data_stored":882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3657728,"data_stored":893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:12.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:12.311 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:12.314 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:12.323 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2021-08-21T15:03:11.494888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33694,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":947,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":59248,"kb_used_data":7968,"kb_used_omap":0,"kb_used_meta":51264,"kb_avail":188684432,"statfs":{"total":193273528320,"available":193212858368,"internally_reserved":0,"allocated":8159232,"data_stored":149480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52494336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-312,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004065"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6796,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:09.692648+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:09.692648+0000","last_peered":"2021-08-21T15:03:09.692648+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:09.692648+0000","last_undegraded":"2021-08-21T15:03:09.692648+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5952,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":947,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":947},{"shards":"2,3","objects":2976}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33694,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":947,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":267251712,"data_stored":412879,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247828,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673002,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29608,"kb_used_data":4000,"kb_used_omap":0,"kb_used_meta":25600,"kb_avail":94342232,"statfs":{"total":96636764160,"available":96606445568,"internally_reserved":0,"allocated":4096000,"data_stored":74585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26214400},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52100000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.48899999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.434}]}]},{"osd":0,"up_from":9,"seq":38654705707,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29640,"kb_used_data":3968,"kb_used_omap":0,"kb_used_meta":25664,"kb_avail":94342200,"statfs":{"total":96636764160,"available":96606412800,"internally_reserved":0,"allocated":4063232,"data_stored":74895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26279936},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.377},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.48499999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3825664,"data_stored":934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3870720,"data_stored":945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:12.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:12.636 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:12.637 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":192,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":60669952,"bytes_avail":193212858368,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":33694,"misplaced_total":63330,"misplaced_ratio":0.53203852834359699,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":12,"recovering_keys_per_sec":0,"num_objects_recovered":156,"num_bytes_recovered":156,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0057544056326150894,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0057544056326150894,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:12.651 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:12.652 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:12.889 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:12.894 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:12.900 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2021-08-21T15:03:11.494888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33694,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":947,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":59248,"kb_used_data":7968,"kb_used_omap":0,"kb_used_meta":51264,"kb_avail":188684432,"statfs":{"total":193273528320,"available":193212858368,"internally_reserved":0,"allocated":8159232,"data_stored":149480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52494336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-312,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004065"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6796,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:09.692648+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:09.692648+0000","last_peered":"2021-08-21T15:03:09.692648+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:09.692648+0000","last_undegraded":"2021-08-21T15:03:09.692648+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5952,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":947,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":947},{"shards":"2,3","objects":2976}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33694,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":947,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":267251712,"data_stored":412879,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247828,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673002,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29608,"kb_used_data":4000,"kb_used_omap":0,"kb_used_meta":25600,"kb_avail":94342232,"statfs":{"total":96636764160,"available":96606445568,"internally_reserved":0,"allocated":4096000,"data_stored":74585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26214400},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:02:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.753,"15min":0.753},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54100000000000004,"15min":0.55000000000000004},"min":{"1min":0.28699999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52100000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.48899999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.434}]}]},{"osd":0,"up_from":9,"seq":38654705707,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29640,"kb_used_data":3968,"kb_used_omap":0,"kb_used_meta":25664,"kb_avail":94342200,"statfs":{"total":96636764160,"available":96606412800,"internally_reserved":0,"allocated":4063232,"data_stored":74895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26279936},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.377},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.48499999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3825664,"data_stored":934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3870720,"data_stored":945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:15.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:16.162 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:16.165 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:16.175 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2021-08-21T15:03:15.496184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33590,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":59856,"kb_used_data":8384,"kb_used_omap":0,"kb_used_meta":51456,"kb_avail":188683824,"statfs":{"total":193273528320,"available":193212235776,"internally_reserved":0,"allocated":8585216,"data_stored":149584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52690944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-312,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004048"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6952,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:14.667727+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:14.667727+0000","last_peered":"2021-08-21T15:03:14.667727+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:14.667727+0000","last_undegraded":"2021-08-21T15:03:14.667727+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5848,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":999},{"shards":"2,3","objects":2924}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33590,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":267677696,"data_stored":412983,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247829,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673003,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29944,"kb_used_data":4208,"kb_used_omap":0,"kb_used_meta":25728,"kb_avail":94341896,"statfs":{"total":96636764160,"available":96606101504,"internally_reserved":0,"allocated":4308992,"data_stored":74637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26345472},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.46000000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.53200000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.67600000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29912,"kb_used_data":4176,"kb_used_omap":0,"kb_used_meta":25728,"kb_avail":94341928,"statfs":{"total":96636764160,"available":96606134272,"internally_reserved":0,"allocated":4276224,"data_stored":74947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26345472},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.39800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4038656,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4083712,"data_stored":997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:16.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:16.490 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:16.491 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":196,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":61292544,"bytes_avail":193212235776,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":33590,"misplaced_total":63330,"misplaced_ratio":0.53039633664929731,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":12,"recovering_keys_per_sec":0,"num_objects_recovered":156,"num_bytes_recovered":156,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0060887401923537254,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0060887401923537254,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:16.503 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:16.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:16.781 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:16.784 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:16.793 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2021-08-21T15:03:15.496184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33590,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":59856,"kb_used_data":8384,"kb_used_omap":0,"kb_used_meta":51456,"kb_avail":188683824,"statfs":{"total":193273528320,"available":193212235776,"internally_reserved":0,"allocated":8585216,"data_stored":149584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52690944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-312,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004048"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6952,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:14.667727+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:14.667727+0000","last_peered":"2021-08-21T15:03:14.667727+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:14.667727+0000","last_undegraded":"2021-08-21T15:03:14.667727+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5848,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":999},{"shards":"2,3","objects":2924}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33590,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":267677696,"data_stored":412983,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247829,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673003,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29944,"kb_used_data":4208,"kb_used_omap":0,"kb_used_meta":25728,"kb_avail":94341896,"statfs":{"total":96636764160,"available":96606101504,"internally_reserved":0,"allocated":4308992,"data_stored":74637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26345472},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.46000000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.53200000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.67600000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29912,"kb_used_data":4176,"kb_used_omap":0,"kb_used_meta":25728,"kb_avail":94341928,"statfs":{"total":96636764160,"available":96606134272,"internally_reserved":0,"allocated":4276224,"data_stored":74947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26345472},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:02:14 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.48999999999999999,"15min":0.498},"min":{"1min":0.317,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.57899999999999996,"15min":0.59999999999999998},"min":{"1min":0.312,"5min":0.255,"15min":0.255},"max":{"1min":0.58599999999999997,"5min":1.821,"15min":1.821},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.39800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4038656,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4083712,"data_stored":997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:19.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:20.048 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:20.058 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:20.060 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2021-08-21T15:03:19.497561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33590,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":60396,"kb_used_data":8796,"kb_used_omap":0,"kb_used_meta":51584,"kb_avail":188683284,"statfs":{"total":193273528320,"available":193211682816,"internally_reserved":0,"allocated":9007104,"data_stored":149687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52822016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003955"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6952,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:14.667727+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:14.667727+0000","last_peered":"2021-08-21T15:03:14.667727+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:14.667727+0000","last_undegraded":"2021-08-21T15:03:14.667727+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5848,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":999},{"shards":"2,3","objects":2924}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33590,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268103680,"data_stored":413087,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247829,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673004,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30216,"kb_used_data":4416,"kb_used_omap":0,"kb_used_meta":25792,"kb_avail":94341624,"statfs":{"total":96636764160,"available":96605822976,"internally_reserved":0,"allocated":4521984,"data_stored":74689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26411008},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.58399999999999996}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.44400000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705709,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30180,"kb_used_data":4380,"kb_used_omap":0,"kb_used_meta":25792,"kb_avail":94341660,"statfs":{"total":96636764160,"available":96605859840,"internally_reserved":0,"allocated":4485120,"data_stored":74998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26411008},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.37},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.42699999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53500000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.57699999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4251648,"data_stored":1038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:20.061 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:20.361 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:20.361 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":200,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":61845504,"bytes_avail":193211682816,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":33590,"misplaced_total":63330,"misplaced_ratio":0.53039633664929731,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0064230742864310741,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0064230742864310741,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:20.373 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:20.374 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:20.662 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:20.663 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:20.672 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2021-08-21T15:03:19.497561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33590,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":60396,"kb_used_data":8796,"kb_used_omap":0,"kb_used_meta":51584,"kb_avail":188683284,"statfs":{"total":193273528320,"available":193211682816,"internally_reserved":0,"allocated":9007104,"data_stored":149687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52822016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003955"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":6952,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:14.667727+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:14.667727+0000","last_peered":"2021-08-21T15:03:14.667727+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:14.667727+0000","last_undegraded":"2021-08-21T15:03:14.667727+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5848,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":999},{"shards":"2,3","objects":2924}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33590,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268103680,"data_stored":413087,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247829,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673004,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30216,"kb_used_data":4416,"kb_used_omap":0,"kb_used_meta":25792,"kb_avail":94341624,"statfs":{"total":96636764160,"available":96605822976,"internally_reserved":0,"allocated":4521984,"data_stored":74689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26411008},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.58399999999999996}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.44400000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705709,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30180,"kb_used_data":4380,"kb_used_omap":0,"kb_used_meta":25792,"kb_avail":94341660,"statfs":{"total":96636764160,"available":96605859840,"internally_reserved":0,"allocated":4485120,"data_stored":74998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26411008},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.37},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.42699999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53500000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.57699999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4251648,"data_stored":1038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:23.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:23.930 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:23.932 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:23.941 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2021-08-21T15:03:23.498747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33486,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":1051,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":60664,"kb_used_data":9000,"kb_used_omap":0,"kb_used_meta":51648,"kb_avail":188683016,"statfs":{"total":193273528320,"available":193211408384,"internally_reserved":0,"allocated":9216000,"data_stored":149738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52887552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003858"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7108,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:19.665275+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:19.665275+0000","last_peered":"2021-08-21T15:03:19.665275+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:19.665275+0000","last_undegraded":"2021-08-21T15:03:19.665275+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5744,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":1051,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1051},{"shards":"2,3","objects":2872}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33486,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":1051,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268316672,"data_stored":413139,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673004,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30216,"kb_used_data":4416,"kb_used_omap":0,"kb_used_meta":25792,"kb_avail":94341624,"statfs":{"total":96636764160,"available":96605822976,"internally_reserved":0,"allocated":4521984,"data_stored":74689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26411008},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.58399999999999996}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.44400000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30448,"kb_used_data":4584,"kb_used_omap":0,"kb_used_meta":25856,"kb_avail":94341392,"statfs":{"total":96636764160,"available":96605585408,"internally_reserved":0,"allocated":4694016,"data_stored":75049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26476544},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.39600000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4464640,"data_stored":1090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:23.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:24.231 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:24.231 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":204,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":62119936,"bytes_avail":193211408384,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":33486,"misplaced_total":63330,"misplaced_ratio":0.52875414495499762,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0064230742864310741,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0064230742864310741,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:24.248 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:24.249 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:24.502 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:24.504 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:24.515 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2021-08-21T15:03:23.498747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33486,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":1051,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":60664,"kb_used_data":9000,"kb_used_omap":0,"kb_used_meta":51648,"kb_avail":188683016,"statfs":{"total":193273528320,"available":193211408384,"internally_reserved":0,"allocated":9216000,"data_stored":149738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52887552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003858"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7108,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:19.665275+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:19.665275+0000","last_peered":"2021-08-21T15:03:19.665275+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:19.665275+0000","last_undegraded":"2021-08-21T15:03:19.665275+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5744,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":1051,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1051},{"shards":"2,3","objects":2872}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33486,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":1051,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268316672,"data_stored":413139,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673004,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30216,"kb_used_data":4416,"kb_used_omap":0,"kb_used_meta":25792,"kb_avail":94341624,"statfs":{"total":96636764160,"available":96605822976,"internally_reserved":0,"allocated":4521984,"data_stored":74689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26411008},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.58399999999999996}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.44400000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30448,"kb_used_data":4584,"kb_used_omap":0,"kb_used_meta":25856,"kb_avail":94341392,"statfs":{"total":96636764160,"available":96605585408,"internally_reserved":0,"allocated":4694016,"data_stored":75049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26476544},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.39600000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4464640,"data_stored":1090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:27.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:27.793 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:27.796 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:27.805 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2021-08-21T15:03:27.500090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33382,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":1103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":60936,"kb_used_data":9208,"kb_used_omap":0,"kb_used_meta":51712,"kb_avail":188682744,"statfs":{"total":193273528320,"available":193211129856,"internally_reserved":0,"allocated":9428992,"data_stored":149790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52953088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003905"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7264,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:24.659191+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:24.659191+0000","last_peered":"2021-08-21T15:03:24.659191+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:24.659191+0000","last_undegraded":"2021-08-21T15:03:24.659191+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5640,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":1103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1103},{"shards":"2,3","objects":2820}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33382,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":1103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268529664,"data_stored":413191,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247831,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673005,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30488,"kb_used_data":4624,"kb_used_omap":0,"kb_used_meta":25856,"kb_avail":94341352,"statfs":{"total":96636764160,"available":96605544448,"internally_reserved":0,"allocated":4734976,"data_stored":74741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26476544},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.34699999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.44700000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.316},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.52300000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30448,"kb_used_data":4584,"kb_used_omap":0,"kb_used_meta":25856,"kb_avail":94341392,"statfs":{"total":96636764160,"available":96605585408,"internally_reserved":0,"allocated":4694016,"data_stored":75049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26476544},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.39600000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4464640,"data_stored":1090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4509696,"data_stored":1101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:27.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:28.107 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:28.108 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":208,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":62398464,"bytes_avail":193211129856,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":33382,"misplaced_total":63330,"misplaced_ratio":0.52711195326069793,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0067574083805084229,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0067574083805084229,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:28.122 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:28.123 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:28.376 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:28.378 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:28.399 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2021-08-21T15:03:27.500090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33382,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":1103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":60936,"kb_used_data":9208,"kb_used_omap":0,"kb_used_meta":51712,"kb_avail":188682744,"statfs":{"total":193273528320,"available":193211129856,"internally_reserved":0,"allocated":9428992,"data_stored":149790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52953088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003905"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7264,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:24.659191+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:24.659191+0000","last_peered":"2021-08-21T15:03:24.659191+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:24.659191+0000","last_undegraded":"2021-08-21T15:03:24.659191+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5640,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":1103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1103},{"shards":"2,3","objects":2820}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33382,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1103,"num_bytes_recovered":1103,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268529664,"data_stored":413191,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247831,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673005,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30488,"kb_used_data":4624,"kb_used_omap":0,"kb_used_meta":25856,"kb_avail":94341352,"statfs":{"total":96636764160,"available":96605544448,"internally_reserved":0,"allocated":4734976,"data_stored":74741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26476544},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.34699999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.44700000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.316},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.52300000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30448,"kb_used_data":4584,"kb_used_omap":0,"kb_used_meta":25856,"kb_avail":94341392,"statfs":{"total":96636764160,"available":96605585408,"internally_reserved":0,"allocated":4694016,"data_stored":75049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26476544},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.39600000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4464640,"data_stored":1090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4509696,"data_stored":1101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:31.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:31.668 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:31.668 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:31.675 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2021-08-21T15:03:31.500950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33276,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":1156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":61612,"kb_used_data":9628,"kb_used_omap":0,"kb_used_meta":51968,"kb_avail":188682068,"statfs":{"total":193273528320,"available":193210437632,"internally_reserved":0,"allocated":9859072,"data_stored":149895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53215232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-314,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003388"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7423,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:29.741595+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:29.741595+0000","last_peered":"2021-08-21T15:03:29.741595+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:29.741595+0000","last_undegraded":"2021-08-21T15:03:29.741595+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5534,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":1156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1156},{"shards":"2,3","objects":2767}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33276,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":1156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268955648,"data_stored":413295,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247832,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673006,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30828,"kb_used_data":4836,"kb_used_omap":0,"kb_used_meta":25984,"kb_avail":94341012,"statfs":{"total":96636764160,"available":96605196288,"internally_reserved":0,"allocated":4952064,"data_stored":74794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26607616},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.311}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.41699999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.44600000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705711,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30784,"kb_used_data":4792,"kb_used_omap":0,"kb_used_meta":25984,"kb_avail":94341056,"statfs":{"total":96636764160,"available":96605241344,"internally_reserved":0,"allocated":4907008,"data_stored":75101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26607616},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.497},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.433}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.56599999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.76400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4677632,"data_stored":1142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4722688,"data_stored":1153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:31.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:31.976 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:31.977 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":212,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":63090688,"bytes_avail":193210437632,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":33382,"misplaced_total":63330,"misplaced_ratio":0.52711195326069793,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0070917424745857716,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0070917424745857716,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:31.988 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:31.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:32.237 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:32.239 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:32.247 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2021-08-21T15:03:31.500950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33276,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":1156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":61612,"kb_used_data":9628,"kb_used_omap":0,"kb_used_meta":51968,"kb_avail":188682068,"statfs":{"total":193273528320,"available":193210437632,"internally_reserved":0,"allocated":9859072,"data_stored":149895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53215232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-314,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003388"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7423,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:29.741595+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:29.741595+0000","last_peered":"2021-08-21T15:03:29.741595+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:29.741595+0000","last_undegraded":"2021-08-21T15:03:29.741595+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5534,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":1156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1156},{"shards":"2,3","objects":2767}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33276,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":1156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268955648,"data_stored":413295,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247832,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673006,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30828,"kb_used_data":4836,"kb_used_omap":0,"kb_used_meta":25984,"kb_avail":94341012,"statfs":{"total":96636764160,"available":96605196288,"internally_reserved":0,"allocated":4952064,"data_stored":74794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26607616},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.311}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.41699999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.44600000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705711,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30784,"kb_used_data":4792,"kb_used_omap":0,"kb_used_meta":25984,"kb_avail":94341056,"statfs":{"total":96636764160,"available":96605241344,"internally_reserved":0,"allocated":4907008,"data_stored":75101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26607616},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.497},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.433}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.56599999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.76400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4677632,"data_stored":1142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4722688,"data_stored":1153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:35.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:35.544 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:35.545 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:35.553 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2021-08-21T15:03:35.501879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33172,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":1208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":62152,"kb_used_data":10040,"kb_used_omap":0,"kb_used_meta":52096,"kb_avail":188681528,"statfs":{"total":193273528320,"available":193209884672,"internally_reserved":0,"allocated":10280960,"data_stored":149998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53346304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-314,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003131"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7579,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:34.711855+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:34.711855+0000","last_peered":"2021-08-21T15:03:34.711855+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:34.711855+0000","last_undegraded":"2021-08-21T15:03:34.711855+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5430,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":1208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1208},{"shards":"2,3","objects":2715}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33172,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":1208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":269385728,"data_stored":413400,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247833,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673007,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31096,"kb_used_data":5040,"kb_used_omap":0,"kb_used_meta":26048,"kb_avail":94340744,"statfs":{"total":96636764160,"available":96604921856,"internally_reserved":0,"allocated":5160960,"data_stored":74845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26673152},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.434}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.56100000000000005}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.621},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.60899999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31056,"kb_used_data":5000,"kb_used_omap":0,"kb_used_meta":26048,"kb_avail":94340784,"statfs":{"total":96636764160,"available":96604962816,"internally_reserved":0,"allocated":5120000,"data_stored":75153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26673152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.314}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.45900000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4894720,"data_stored":1195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4935680,"data_stored":1205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:35.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:35.879 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:35.880 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":216,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":63369216,"bytes_avail":193210159104,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":33276,"misplaced_total":63330,"misplaced_ratio":0.52543818095689243,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0070917424745857716,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 6h)","progress":0.0070917424745857716,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:35.891 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:35.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:36.164 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:36.166 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:36.167 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2021-08-21T15:03:35.501879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33172,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":1208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":62152,"kb_used_data":10040,"kb_used_omap":0,"kb_used_meta":52096,"kb_avail":188681528,"statfs":{"total":193273528320,"available":193209884672,"internally_reserved":0,"allocated":10280960,"data_stored":149998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53346304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-314,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003131"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7579,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:34.711855+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:34.711855+0000","last_peered":"2021-08-21T15:03:34.711855+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:34.711855+0000","last_undegraded":"2021-08-21T15:03:34.711855+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5430,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":1208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1208},{"shards":"2,3","objects":2715}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33172,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":1208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":269385728,"data_stored":413400,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247833,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673007,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31096,"kb_used_data":5040,"kb_used_omap":0,"kb_used_meta":26048,"kb_avail":94340744,"statfs":{"total":96636764160,"available":96604921856,"internally_reserved":0,"allocated":5160960,"data_stored":74845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26673152},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.434}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.56100000000000005}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.621},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.60899999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31056,"kb_used_data":5000,"kb_used_omap":0,"kb_used_meta":26048,"kb_avail":94340784,"statfs":{"total":96636764160,"available":96604962816,"internally_reserved":0,"allocated":5120000,"data_stored":75153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26673152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.314}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.45900000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4894720,"data_stored":1195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4935680,"data_stored":1205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:39.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:39.429 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:39.431 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:39.441 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2021-08-21T15:03:37.502412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33172,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":1208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":62152,"kb_used_data":10040,"kb_used_omap":0,"kb_used_meta":52096,"kb_avail":188681528,"statfs":{"total":193273528320,"available":193209884672,"internally_reserved":0,"allocated":10280960,"data_stored":149998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53346304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-210,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003028"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7579,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:34.711855+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:34.711855+0000","last_peered":"2021-08-21T15:03:34.711855+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:34.711855+0000","last_undegraded":"2021-08-21T15:03:34.711855+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5430,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":1208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1208},{"shards":"2,3","objects":2715}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33172,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":1208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":269385728,"data_stored":413400,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247833,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673007,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31096,"kb_used_data":5040,"kb_used_omap":0,"kb_used_meta":26048,"kb_avail":94340744,"statfs":{"total":96636764160,"available":96604921856,"internally_reserved":0,"allocated":5160960,"data_stored":74845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26673152},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.434}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.56100000000000005}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.621},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.60899999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31056,"kb_used_data":5000,"kb_used_omap":0,"kb_used_meta":26048,"kb_avail":94340784,"statfs":{"total":96636764160,"available":96604962816,"internally_reserved":0,"allocated":5120000,"data_stored":75153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26673152},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.314}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.45900000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4894720,"data_stored":1195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4935680,"data_stored":1205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:39.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:39.753 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:39.754 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":220,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":63643648,"bytes_avail":193209884672,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":33172,"misplaced_total":63330,"misplaced_ratio":0.52379598926259274,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 5h)","progress":0.0074325064197182655,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 5h)","progress":0.0074325064197182655,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:39.784 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:39.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:40.062 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:40.066 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:40.075 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2021-08-21T15:03:39.503014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33172,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":1208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":62704,"kb_used_data":10464,"kb_used_omap":0,"kb_used_meta":52224,"kb_avail":188680976,"statfs":{"total":193273528320,"available":193209319424,"internally_reserved":0,"allocated":10715136,"data_stored":150104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53477376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-210,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002923"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7579,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:34.711855+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:34.711855+0000","last_peered":"2021-08-21T15:03:34.711855+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:34.711855+0000","last_undegraded":"2021-08-21T15:03:34.711855+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5430,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":1208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1208},{"shards":"2,3","objects":2715}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33172,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":1208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":269815808,"data_stored":413505,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247833,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673008,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":5252,"kb_used_omap":0,"kb_used_meta":26112,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":5378048,"data_stored":74898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26738688},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68600000000000005}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.752},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.61599999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31332,"kb_used_data":5212,"kb_used_omap":0,"kb_used_meta":26112,"kb_avail":94340508,"statfs":{"total":96636764160,"available":96604680192,"internally_reserved":0,"allocated":5337088,"data_stored":75206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26738688},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.436},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.28000000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.35399999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.313},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.40300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5107712,"data_stored":1247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5152768,"data_stored":1258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:43.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:43.338 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:43.341 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:43.350 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2021-08-21T15:03:41.503647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33068,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1260,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":62704,"kb_used_data":10464,"kb_used_omap":0,"kb_used_meta":52224,"kb_avail":188680976,"statfs":{"total":193273528320,"available":193209319424,"internally_reserved":0,"allocated":10715136,"data_stored":150104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53477376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-314,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003098"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7735,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:39.677653+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:39.677653+0000","last_peered":"2021-08-21T15:03:39.677653+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:39.677653+0000","last_undegraded":"2021-08-21T15:03:39.677653+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5326,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1260,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1260},{"shards":"2,3","objects":2663}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33068,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1260,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":269815808,"data_stored":413505,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117015,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247834,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673008,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":5252,"kb_used_omap":0,"kb_used_meta":26112,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":5378048,"data_stored":74898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26738688},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68600000000000005}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.752},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.61599999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31332,"kb_used_data":5212,"kb_used_omap":0,"kb_used_meta":26112,"kb_avail":94340508,"statfs":{"total":96636764160,"available":96604680192,"internally_reserved":0,"allocated":5337088,"data_stored":75206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26738688},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.436},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.28000000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:02:37 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.35399999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.313},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.40300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5107712,"data_stored":1247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5152768,"data_stored":1258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:43.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:43.659 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:43.660 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":223,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":64208896,"bytes_avail":193209319424,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":33068,"misplaced_total":63330,"misplaced_ratio":0.52215379756829305,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":13,"recovering_keys_per_sec":0,"num_objects_recovered":157,"num_bytes_recovered":157,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 5h)","progress":0.0077668405137956142,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 5h)","progress":0.0077668405137956142,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:43.672 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:43.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:43.927 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:43.929 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:43.943 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2021-08-21T15:03:43.504422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33068,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1260,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":63036,"kb_used_data":10668,"kb_used_omap":0,"kb_used_meta":52352,"kb_avail":188680644,"statfs":{"total":193273528320,"available":193208979456,"internally_reserved":0,"allocated":10924032,"data_stored":150155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53608448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003472"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7735,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:39.677653+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:39.677653+0000","last_peered":"2021-08-21T15:03:39.677653+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:39.677653+0000","last_undegraded":"2021-08-21T15:03:39.677653+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5326,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1260,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1260},{"shards":"2,3","objects":2663}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":33068,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1260,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270032896,"data_stored":413558,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117015,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247834,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673008,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":5252,"kb_used_omap":0,"kb_used_meta":26112,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":5378048,"data_stored":74898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26738688},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68600000000000005}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.752},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.61599999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":5416,"kb_used_omap":0,"kb_used_meta":26240,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":5545984,"data_stored":75257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26869760},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.53900000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.5},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.57899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":1300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5152768,"data_stored":1258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:46.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:47.200 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:47.202 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:47.213 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2021-08-21T15:03:45.504948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32962,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":1313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":63376,"kb_used_data":10880,"kb_used_omap":0,"kb_used_meta":52480,"kb_avail":188680304,"statfs":{"total":193273528320,"available":193208631296,"internally_reserved":0,"allocated":11141120,"data_stored":150208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53739520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-314,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003548"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7894,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:44.741635+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:44.741635+0000","last_peered":"2021-08-21T15:03:44.741635+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:44.741635+0000","last_undegraded":"2021-08-21T15:03:44.741635+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5220,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":1313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1313},{"shards":"2,3","objects":2610}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32962,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":1313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270245888,"data_stored":413610,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247835,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673009,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":5464,"kb_used_omap":0,"kb_used_meta":26240,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":5595136,"data_stored":74951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26869760},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68600000000000005}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.752},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.61599999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":5416,"kb_used_omap":0,"kb_used_meta":26240,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":5545984,"data_stored":75257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26869760},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.53900000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.5},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.57899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":1300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5365760,"data_stored":1310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:47.216 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:47.525 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:47.525 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":227,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":64897024,"bytes_avail":193208631296,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":32962,"misplaced_total":63330,"misplaced_ratio":0.52048002526448756,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":13,"recovering_keys_per_sec":0,"num_objects_recovered":157,"num_bytes_recovered":157,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 5h)","progress":0.008101174607872963,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 5h)","progress":0.008101174607872963,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:47.539 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:47.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:47.798 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:47.800 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:47.809 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2021-08-21T15:03:47.505351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32962,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":1313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":63376,"kb_used_data":10880,"kb_used_omap":0,"kb_used_meta":52480,"kb_avail":188680304,"statfs":{"total":193273528320,"available":193208631296,"internally_reserved":0,"allocated":11141120,"data_stored":150208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53739520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-210,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003472"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7894,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:44.741635+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:44.741635+0000","last_peered":"2021-08-21T15:03:44.741635+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:44.741635+0000","last_undegraded":"2021-08-21T15:03:44.741635+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5220,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":1313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1313},{"shards":"2,3","objects":2610}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32962,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":1313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270245888,"data_stored":413610,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247835,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673009,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":5464,"kb_used_omap":0,"kb_used_meta":26240,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":5595136,"data_stored":74951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26869760},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68600000000000005}]},{"osd":2,"last update":"Sat Aug 21 15:02:39 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.34999999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.752},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.61599999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31664,"kb_used_data":5416,"kb_used_omap":0,"kb_used_meta":26240,"kb_avail":94340176,"statfs":{"total":96636764160,"available":96604340224,"internally_reserved":0,"allocated":5545984,"data_stored":75257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26869760},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.53900000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.5},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.57899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5324800,"data_stored":1300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5365760,"data_stored":1310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:50.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:51.057 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:51.059 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:51.068 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2021-08-21T15:03:49.505975+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32962,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":1313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":63924,"kb_used_data":11300,"kb_used_omap":0,"kb_used_meta":52608,"kb_avail":188679756,"statfs":{"total":193273528320,"available":193208070144,"internally_reserved":0,"allocated":11571200,"data_stored":150313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53870592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-210,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003563"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":7894,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:44.741635+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:44.741635+0000","last_peered":"2021-08-21T15:03:44.741635+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:44.741635+0000","last_undegraded":"2021-08-21T15:03:44.741635+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5220,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":1313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1313},{"shards":"2,3","objects":2610}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32962,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":1313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270671872,"data_stored":413714,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247835,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673010,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":5672,"kb_used_omap":0,"kb_used_meta":26304,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":5808128,"data_stored":75003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26935296},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.56399999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.49299999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705715,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":5628,"kb_used_omap":0,"kb_used_meta":26304,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":5763072,"data_stored":75310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26935296},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.38300000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.57399999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.65700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5537792,"data_stored":1352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5578752,"data_stored":1362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:51.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:51.377 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:51.377 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":231,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":65458176,"bytes_avail":193208070144,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":32962,"misplaced_total":63330,"misplaced_ratio":0.52048002526448756,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 5h)","progress":0.0084419380873441696,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 5h)","progress":0.0084419380873441696,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:51.390 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:51.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:51.641 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:51.643 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:51.652 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2021-08-21T15:03:51.506543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32858,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1365,"num_bytes_recovered":1365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":63924,"kb_used_data":11300,"kb_used_omap":0,"kb_used_meta":52608,"kb_avail":188679756,"statfs":{"total":193273528320,"available":193208070144,"internally_reserved":0,"allocated":11571200,"data_stored":150313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53870592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-314,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":157,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003529"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8050,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:49.701306+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:49.701306+0000","last_peered":"2021-08-21T15:03:49.701306+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:49.701306+0000","last_undegraded":"2021-08-21T15:03:49.701306+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5116,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1365,"num_bytes_recovered":1365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1365},{"shards":"2,3","objects":2558}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32858,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1365,"num_bytes_recovered":1365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270671872,"data_stored":413714,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117017,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247836,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673010,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":5672,"kb_used_omap":0,"kb_used_meta":26304,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":5808128,"data_stored":75003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26935296},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.56399999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.49299999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705715,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31940,"kb_used_data":5628,"kb_used_omap":0,"kb_used_meta":26304,"kb_avail":94339900,"statfs":{"total":96636764160,"available":96604057600,"internally_reserved":0,"allocated":5763072,"data_stored":75310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26935296},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.38300000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.57399999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.65700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5537792,"data_stored":1352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5578752,"data_stored":1362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:54.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:54.913 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:54.917 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:54.931 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2021-08-21T15:03:53.506990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32858,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1365,"num_bytes_recovered":1365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":64200,"kb_used_data":11512,"kb_used_omap":0,"kb_used_meta":52672,"kb_avail":188679480,"statfs":{"total":193273528320,"available":193207787520,"internally_reserved":0,"allocated":11788288,"data_stored":150366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53936128},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-210,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003342"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8050,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:49.701306+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:49.701306+0000","last_peered":"2021-08-21T15:03:49.701306+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:49.701306+0000","last_undegraded":"2021-08-21T15:03:49.701306+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5116,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1365,"num_bytes_recovered":1365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1365},{"shards":"2,3","objects":2558}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32858,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1365,"num_bytes_recovered":1365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270884864,"data_stored":413766,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117017,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247836,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673010,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":5672,"kb_used_omap":0,"kb_used_meta":26304,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":5808128,"data_stored":75003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26935296},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.56399999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.49299999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":5840,"kb_used_omap":0,"kb_used_meta":26368,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":5980160,"data_stored":75363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27000832},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5750784,"data_stored":1404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5578752,"data_stored":1362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:54.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:55.232 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:55.232 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":235,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":65740800,"bytes_avail":193207787520,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":32858,"misplaced_total":63330,"misplaced_ratio":0.51883783357018787,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (2m)\n [............................] (remaining: 5h)","progress":0.0084419380873441696,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (2m)\n [............................] (remaining: 5h)","progress":0.0084419380873441696,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (2m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:55.250 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:55.251 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:55.503 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:55.507 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:55.518 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2021-08-21T15:03:53.506990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32858,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1365,"num_bytes_recovered":1365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":64200,"kb_used_data":11512,"kb_used_omap":0,"kb_used_meta":52672,"kb_avail":188679480,"statfs":{"total":193273528320,"available":193207787520,"internally_reserved":0,"allocated":11788288,"data_stored":150366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53936128},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-210,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003342"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8050,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:49.701306+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:49.701306+0000","last_peered":"2021-08-21T15:03:49.701306+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:49.701306+0000","last_undegraded":"2021-08-21T15:03:49.701306+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5116,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1365,"num_bytes_recovered":1365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1365},{"shards":"2,3","objects":2558}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32858,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1365,"num_bytes_recovered":1365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270884864,"data_stored":413766,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117017,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247836,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673010,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31984,"kb_used_data":5672,"kb_used_omap":0,"kb_used_meta":26304,"kb_avail":94339856,"statfs":{"total":96636764160,"available":96604012544,"internally_reserved":0,"allocated":5808128,"data_stored":75003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26935296},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.56399999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:02:51 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.49299999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":5840,"kb_used_omap":0,"kb_used_meta":26368,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":5980160,"data_stored":75363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27000832},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5750784,"data_stored":1404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5578752,"data_stored":1362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:58.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:58.783 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:58.785 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:58.796 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2021-08-21T15:03:57.508041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32752,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":1418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":64472,"kb_used_data":11720,"kb_used_omap":0,"kb_used_meta":52736,"kb_avail":188679208,"statfs":{"total":193273528320,"available":193207508992,"internally_reserved":0,"allocated":12001280,"data_stored":150418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54001664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-210,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003093"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8209,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:54.734639+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:54.734639+0000","last_peered":"2021-08-21T15:03:54.734639+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:54.734639+0000","last_undegraded":"2021-08-21T15:03:54.734639+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5010,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":1418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1418},{"shards":"2,3","objects":2505}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32752,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":1418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":271101952,"data_stored":413819,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247837,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673011,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":5880,"kb_used_omap":0,"kb_used_meta":26368,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":6021120,"data_stored":75055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27000832},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54800000000000004}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.53800000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":5840,"kb_used_omap":0,"kb_used_meta":26368,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":5980160,"data_stored":75363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27000832},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5750784,"data_stored":1404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5795840,"data_stored":1415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:03:58.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:03:59.102 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:59.102 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":239,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":66019328,"bytes_avail":193207508992,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":32752,"misplaced_total":63330,"misplaced_ratio":0.51716406126638248,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.0087762726470828056,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.0087762726470828056,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (2m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:03:59.112 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:03:59.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:03:59.367 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:03:59.370 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:03:59.380 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2021-08-21T15:03:57.508041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32752,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":1418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":64472,"kb_used_data":11720,"kb_used_omap":0,"kb_used_meta":52736,"kb_avail":188679208,"statfs":{"total":193273528320,"available":193207508992,"internally_reserved":0,"allocated":12001280,"data_stored":150418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54001664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-210,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003093"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8209,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:54.734639+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:54.734639+0000","last_peered":"2021-08-21T15:03:54.734639+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:54.734639+0000","last_undegraded":"2021-08-21T15:03:54.734639+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5010,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":1418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1418},{"shards":"2,3","objects":2505}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32752,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1418,"num_bytes_recovered":1418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":271101952,"data_stored":413819,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247837,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673011,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":5880,"kb_used_omap":0,"kb_used_meta":26368,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":6021120,"data_stored":75055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27000832},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.54800000000000004}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.53800000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":5840,"kb_used_omap":0,"kb_used_meta":26368,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":5980160,"data_stored":75363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27000832},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.59599999999999997}]},{"osd":3,"last update":"Sat Aug 21 15:02:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5750784,"data_stored":1404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5795840,"data_stored":1415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:02.382 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:02.643 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:02.643 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:02.649 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2021-08-21T15:04:01.509224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32646,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1471,"num_bytes_recovered":1471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":65144,"kb_used_data":12136,"kb_used_omap":0,"kb_used_meta":52992,"kb_avail":188678536,"statfs":{"total":193273528320,"available":193206820864,"internally_reserved":0,"allocated":12427264,"data_stored":150522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54263808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-316,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003248"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8368,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:59.752253+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:59.752253+0000","last_peered":"2021-08-21T15:03:59.752253+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:59.752253+0000","last_undegraded":"2021-08-21T15:03:59.752253+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4904,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1471,"num_bytes_recovered":1471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1471},{"shards":"2,3","objects":2452}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32646,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1471,"num_bytes_recovered":1471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":271536128,"data_stored":413925,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117019,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673012,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32592,"kb_used_data":6088,"kb_used_omap":0,"kb_used_meta":26496,"kb_avail":94339248,"statfs":{"total":96636764160,"available":96603389952,"internally_reserved":0,"allocated":6234112,"data_stored":75107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27131904},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51800000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.57299999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.61699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705717,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32552,"kb_used_data":6048,"kb_used_omap":0,"kb_used_meta":26496,"kb_avail":94339288,"statfs":{"total":96636764160,"available":96603430912,"internally_reserved":0,"allocated":6193152,"data_stored":75415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27131904},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.26400000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.313},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.38700000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.34100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5967872,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6012928,"data_stored":1468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:02.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:02.958 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:02.958 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":243,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":66707456,"bytes_avail":193206820864,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":32646,"misplaced_total":63330,"misplaced_ratio":0.51549028896257698,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":13,"recovering_keys_per_sec":0,"num_objects_recovered":158,"num_bytes_recovered":158,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.0091170361265540123,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.0091170361265540123,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:02.975 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:02.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:03.228 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:03.230 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:03.240 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2021-08-21T15:04:01.509224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32646,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1471,"num_bytes_recovered":1471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":65144,"kb_used_data":12136,"kb_used_omap":0,"kb_used_meta":52992,"kb_avail":188678536,"statfs":{"total":193273528320,"available":193206820864,"internally_reserved":0,"allocated":12427264,"data_stored":150522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54263808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-316,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003248"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8368,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:03:59.752253+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:03:59.752253+0000","last_peered":"2021-08-21T15:03:59.752253+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:03:59.752253+0000","last_undegraded":"2021-08-21T15:03:59.752253+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4904,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1471,"num_bytes_recovered":1471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1471},{"shards":"2,3","objects":2452}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32646,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1471,"num_bytes_recovered":1471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":271536128,"data_stored":413925,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117019,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673012,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32592,"kb_used_data":6088,"kb_used_omap":0,"kb_used_meta":26496,"kb_avail":94339248,"statfs":{"total":96636764160,"available":96603389952,"internally_reserved":0,"allocated":6234112,"data_stored":75107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27131904},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51800000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.57299999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.61699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705717,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32552,"kb_used_data":6048,"kb_used_omap":0,"kb_used_meta":26496,"kb_avail":94339288,"statfs":{"total":96636764160,"available":96603430912,"internally_reserved":0,"allocated":6193152,"data_stored":75415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27131904},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.26400000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.313},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.38700000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.34100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5967872,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6012928,"data_stored":1468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:06.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:06.518 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:06.521 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:06.530 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2021-08-21T15:04:05.510332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32542,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":1523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":65696,"kb_used_data":12560,"kb_used_omap":0,"kb_used_meta":53120,"kb_avail":188677984,"statfs":{"total":193273528320,"available":193206255616,"internally_reserved":0,"allocated":12861440,"data_stored":150628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54394880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-316,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003341"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8524,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:04.697395+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:04.697395+0000","last_peered":"2021-08-21T15:04:04.697395+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:04.697395+0000","last_undegraded":"2021-08-21T15:04:04.697395+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4800,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":1523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1523},{"shards":"2,3","objects":2400}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32542,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":1523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":271966208,"data_stored":414030,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247839,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673013,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32864,"kb_used_data":6296,"kb_used_omap":0,"kb_used_meta":26560,"kb_avail":94338976,"statfs":{"total":96636764160,"available":96603111424,"internally_reserved":0,"allocated":6447104,"data_stored":75159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27197440},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52100000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.627},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.34200000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32832,"kb_used_data":6264,"kb_used_omap":0,"kb_used_meta":26560,"kb_avail":94339008,"statfs":{"total":96636764160,"available":96603144192,"internally_reserved":0,"allocated":6414336,"data_stored":75469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27197440},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.39500000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.437}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.48599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6184960,"data_stored":1510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6225920,"data_stored":1520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:06.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:06.864 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:06.865 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":247,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":67272704,"bytes_avail":193206255616,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":32542,"misplaced_total":63330,"misplaced_ratio":0.51384809726827729,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":13,"recovering_keys_per_sec":0,"num_objects_recovered":158,"num_bytes_recovered":158,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.009457799606025219,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.009457799606025219,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:06.879 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:06.880 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:07.147 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:07.149 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:07.150 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2021-08-21T15:04:05.510332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32542,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":1523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":65696,"kb_used_data":12560,"kb_used_omap":0,"kb_used_meta":53120,"kb_avail":188677984,"statfs":{"total":193273528320,"available":193206255616,"internally_reserved":0,"allocated":12861440,"data_stored":150628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54394880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-316,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003341"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8524,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:04.697395+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:04.697395+0000","last_peered":"2021-08-21T15:04:04.697395+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:04.697395+0000","last_undegraded":"2021-08-21T15:04:04.697395+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4800,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":1523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1523},{"shards":"2,3","objects":2400}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32542,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":1523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":271966208,"data_stored":414030,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247839,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673013,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32864,"kb_used_data":6296,"kb_used_omap":0,"kb_used_meta":26560,"kb_avail":94338976,"statfs":{"total":96636764160,"available":96603111424,"internally_reserved":0,"allocated":6447104,"data_stored":75159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27197440},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52100000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.627},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.34200000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32832,"kb_used_data":6264,"kb_used_omap":0,"kb_used_meta":26560,"kb_avail":94339008,"statfs":{"total":96636764160,"available":96603144192,"internally_reserved":0,"allocated":6414336,"data_stored":75469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27197440},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.39500000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.437}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.48599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6184960,"data_stored":1510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6225920,"data_stored":1520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:10.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:10.422 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:10.422 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:10.430 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2021-08-21T15:04:09.511392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32542,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":1523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":66244,"kb_used_data":12980,"kb_used_omap":0,"kb_used_meta":53248,"kb_avail":188677436,"statfs":{"total":193273528320,"available":193205694464,"internally_reserved":0,"allocated":13291520,"data_stored":150733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54525952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-210,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003350"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8524,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:04.697395+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:04.697395+0000","last_peered":"2021-08-21T15:04:04.697395+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:04.697395+0000","last_undegraded":"2021-08-21T15:04:04.697395+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4800,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":1523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1523},{"shards":"2,3","objects":2400}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32542,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":1523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":272400384,"data_stored":414136,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247839,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673014,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33136,"kb_used_data":6504,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":6660096,"data_stored":75211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.60499999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.28799999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.30199999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33108,"kb_used_data":6476,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":94338732,"statfs":{"total":96636764160,"available":96602861568,"internally_reserved":0,"allocated":6631424,"data_stored":75522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.218},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.436}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.255}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.36199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6402048,"data_stored":1563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6443008,"data_stored":1573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:10.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:10.744 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:10.745 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":251,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":67833856,"bytes_avail":193205694464,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":32542,"misplaced_total":63330,"misplaced_ratio":0.51384809726827729,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.009792134165763855,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.009792134165763855,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:10.764 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:10.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:11.036 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:11.038 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:11.048 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2021-08-21T15:04:09.511392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32542,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":1523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":66244,"kb_used_data":12980,"kb_used_omap":0,"kb_used_meta":53248,"kb_avail":188677436,"statfs":{"total":193273528320,"available":193205694464,"internally_reserved":0,"allocated":13291520,"data_stored":150733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54525952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-210,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003350"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8524,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:04.697395+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:04.697395+0000","last_peered":"2021-08-21T15:04:04.697395+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:04.697395+0000","last_undegraded":"2021-08-21T15:04:04.697395+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4800,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":1523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1523},{"shards":"2,3","objects":2400}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32542,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1523,"num_bytes_recovered":1523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":272400384,"data_stored":414136,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247839,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673014,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33136,"kb_used_data":6504,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":6660096,"data_stored":75211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.60499999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.28799999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.30199999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33108,"kb_used_data":6476,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":94338732,"statfs":{"total":96636764160,"available":96602861568,"internally_reserved":0,"allocated":6631424,"data_stored":75522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.218},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.436}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.255}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.36199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6402048,"data_stored":1563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6443008,"data_stored":1573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:14.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:14.314 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:14.323 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2021-08-21T15:04:13.512473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32436,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1576,"num_bytes_recovered":1576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":66584,"kb_used_data":13192,"kb_used_omap":0,"kb_used_meta":53376,"kb_avail":188677096,"statfs":{"total":193273528320,"available":193205346304,"internally_reserved":0,"allocated":13508608,"data_stored":150786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54657024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-210,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003249"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8683,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:09.720291+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:09.720291+0000","last_peered":"2021-08-21T15:04:09.720291+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:09.720291+0000","last_undegraded":"2021-08-21T15:04:09.720291+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4694,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1576,"num_bytes_recovered":1576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1576},{"shards":"2,3","objects":2347}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32436,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1576,"num_bytes_recovered":1576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":272613376,"data_stored":414188,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117021,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247840,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673014,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33136,"kb_used_data":6504,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":6660096,"data_stored":75211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.60499999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.28799999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.30199999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33448,"kb_used_data":6688,"kb_used_omap":0,"kb_used_meta":26752,"kb_avail":94338392,"statfs":{"total":96636764160,"available":96602513408,"internally_reserved":0,"allocated":6848512,"data_stored":75575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27394048},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.52100000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.38},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.59899999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.40999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6615040,"data_stored":1615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6443008,"data_stored":1573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:14.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:14.618 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":254,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":68182016,"bytes_avail":193205346304,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":32436,"misplaced_total":63330,"misplaced_ratio":0.51217432496447179,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.009792134165763855,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.009792134165763855,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:14.635 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:14.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:14.884 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:14.895 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:14.896 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2021-08-21T15:04:13.512473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32436,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1576,"num_bytes_recovered":1576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":66584,"kb_used_data":13192,"kb_used_omap":0,"kb_used_meta":53376,"kb_avail":188677096,"statfs":{"total":193273528320,"available":193205346304,"internally_reserved":0,"allocated":13508608,"data_stored":150786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54657024},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-210,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003249"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8683,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:09.720291+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:09.720291+0000","last_peered":"2021-08-21T15:04:09.720291+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:09.720291+0000","last_undegraded":"2021-08-21T15:04:09.720291+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4694,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1576,"num_bytes_recovered":1576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1576},{"shards":"2,3","objects":2347}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32436,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1576,"num_bytes_recovered":1576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":272613376,"data_stored":414188,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117021,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247840,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673014,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33136,"kb_used_data":6504,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":94338704,"statfs":{"total":96636764160,"available":96602832896,"internally_reserved":0,"allocated":6660096,"data_stored":75211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:03:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.49399999999999999,"15min":0.52100000000000002},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.64400000000000002,"5min":0.753,"15min":0.753},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.51400000000000001,"15min":0.54100000000000004},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.61599999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.60499999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.28799999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.30199999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33448,"kb_used_data":6688,"kb_used_omap":0,"kb_used_meta":26752,"kb_avail":94338392,"statfs":{"total":96636764160,"available":96602513408,"internally_reserved":0,"allocated":6848512,"data_stored":75575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27394048},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.52100000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.38},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.59899999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.40999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6615040,"data_stored":1615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6443008,"data_stored":1573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:17.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:18.161 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:18.164 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:18.173 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2021-08-21T15:04:17.513905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32330,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1629,"num_bytes_recovered":1629,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":71016,"kb_used_data":13400,"kb_used_omap":0,"kb_used_meta":57600,"kb_avail":188672664,"statfs":{"total":193273528320,"available":193200807936,"internally_reserved":0,"allocated":13721600,"data_stored":150838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58982400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003573"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8842,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:14.740034+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:14.740034+0000","last_peered":"2021-08-21T15:04:14.740034+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:14.740034+0000","last_undegraded":"2021-08-21T15:04:14.740034+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4588,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1629,"num_bytes_recovered":1629,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1629},{"shards":"2,3","objects":2294}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32330,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1629,"num_bytes_recovered":1629,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":272830464,"data_stored":414241,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247841,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673015,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37568,"kb_used_data":6712,"kb_used_omap":0,"kb_used_meta":30848,"kb_avail":94334272,"statfs":{"total":96636764160,"available":96598294528,"internally_reserved":0,"allocated":6873088,"data_stored":75263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31588352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.372}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.441},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.51600000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33448,"kb_used_data":6688,"kb_used_omap":0,"kb_used_meta":26752,"kb_avail":94338392,"statfs":{"total":96636764160,"available":96602513408,"internally_reserved":0,"allocated":6848512,"data_stored":75575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27394048},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.52100000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.38},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.59899999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.40999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6615040,"data_stored":1615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6660096,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:18.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:18.485 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:18.485 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":258,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":72720384,"bytes_avail":193200807936,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":32330,"misplaced_total":63330,"misplaced_ratio":0.5105005526606663,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.010132897645235062,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.010132897645235062,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:18.502 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:18.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:18.763 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:18.764 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:18.776 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2021-08-21T15:04:17.513905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32330,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1629,"num_bytes_recovered":1629,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":71016,"kb_used_data":13400,"kb_used_omap":0,"kb_used_meta":57600,"kb_avail":188672664,"statfs":{"total":193273528320,"available":193200807936,"internally_reserved":0,"allocated":13721600,"data_stored":150838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58982400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003573"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8842,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:14.740034+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:14.740034+0000","last_peered":"2021-08-21T15:04:14.740034+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:14.740034+0000","last_undegraded":"2021-08-21T15:04:14.740034+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4588,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1629,"num_bytes_recovered":1629,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1629},{"shards":"2,3","objects":2294}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32330,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1629,"num_bytes_recovered":1629,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":272830464,"data_stored":414241,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247841,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673015,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37568,"kb_used_data":6712,"kb_used_omap":0,"kb_used_meta":30848,"kb_avail":94334272,"statfs":{"total":96636764160,"available":96598294528,"internally_reserved":0,"allocated":6873088,"data_stored":75263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31588352},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.372}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.441},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.51600000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33448,"kb_used_data":6688,"kb_used_omap":0,"kb_used_meta":26752,"kb_avail":94338392,"statfs":{"total":96636764160,"available":96602513408,"internally_reserved":0,"allocated":6848512,"data_stored":75575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27394048},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:03:16 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47599999999999998,"15min":0.49399999999999999},"min":{"1min":0.252,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.624,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.437,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27200000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.60099999999999998,"5min":1.821,"15min":1.821},"last":0.52100000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.38},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.59899999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.40999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6615040,"data_stored":1615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6660096,"data_stored":1626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:21.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:22.058 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:22.064 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:22.075 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2021-08-21T15:04:21.515152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32226,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1681,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":75652,"kb_used_data":13812,"kb_used_omap":0,"kb_used_meta":61824,"kb_avail":188668028,"statfs":{"total":193273528320,"available":193196060672,"internally_reserved":0,"allocated":14143488,"data_stored":150941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63307776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-316,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003759"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8998,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:19.667971+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:19.667971+0000","last_peered":"2021-08-21T15:04:19.667971+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:19.667971+0000","last_undegraded":"2021-08-21T15:04:19.667971+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4484,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1681,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1681},{"shards":"2,3","objects":2242}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32226,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1681,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":273264640,"data_stored":414347,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117023,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247842,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673016,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37840,"kb_used_data":6920,"kb_used_omap":0,"kb_used_meta":30912,"kb_avail":94334000,"statfs":{"total":96636764160,"available":96598016000,"internally_reserved":0,"allocated":7086080,"data_stored":75315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31653888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.372}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.441},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.51600000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705721,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37812,"kb_used_data":6892,"kb_used_omap":0,"kb_used_meta":30912,"kb_avail":94334028,"statfs":{"total":96636764160,"available":96598044672,"internally_reserved":0,"allocated":7057408,"data_stored":75626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31653888},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.35499999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.63700000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.50800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6832128,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6877184,"data_stored":1679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:22.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:22.386 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:22.387 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":262,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":77467648,"bytes_avail":193196060672,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":32226,"misplaced_total":63330,"misplaced_ratio":0.50885836096636661,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":13,"recovering_keys_per_sec":0,"num_objects_recovered":158,"num_bytes_recovered":158,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.010473661124706268,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.010473661124706268,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:22.402 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:22.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:22.656 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:22.657 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:22.669 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2021-08-21T15:04:21.515152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32226,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1681,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":75652,"kb_used_data":13812,"kb_used_omap":0,"kb_used_meta":61824,"kb_avail":188668028,"statfs":{"total":193273528320,"available":193196060672,"internally_reserved":0,"allocated":14143488,"data_stored":150941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63307776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-316,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003759"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":8998,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:19.667971+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:19.667971+0000","last_peered":"2021-08-21T15:04:19.667971+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:19.667971+0000","last_undegraded":"2021-08-21T15:04:19.667971+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4484,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1681,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1681},{"shards":"2,3","objects":2242}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32226,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1681,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":273264640,"data_stored":414347,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117023,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247842,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673016,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37840,"kb_used_data":6920,"kb_used_omap":0,"kb_used_meta":30912,"kb_avail":94334000,"statfs":{"total":96636764160,"available":96598016000,"internally_reserved":0,"allocated":7086080,"data_stored":75315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31653888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.38800000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.372}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.441},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.51600000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705721,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37812,"kb_used_data":6892,"kb_used_omap":0,"kb_used_meta":30912,"kb_avail":94334028,"statfs":{"total":96636764160,"available":96598044672,"internally_reserved":0,"allocated":7057408,"data_stored":75626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31653888},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.35499999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.63700000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.50800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6832128,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6877184,"data_stored":1679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:25.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:25.946 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:25.950 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:25.960 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2021-08-21T15:04:25.516666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32120,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":76204,"kb_used_data":14236,"kb_used_omap":0,"kb_used_meta":61952,"kb_avail":188667476,"statfs":{"total":193273528320,"available":193195495424,"internally_reserved":0,"allocated":14577664,"data_stored":151047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63438848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-316,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004193"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9157,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:24.689400+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:24.689400+0000","last_peered":"2021-08-21T15:04:24.689400+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:24.689400+0000","last_undegraded":"2021-08-21T15:04:24.689400+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4378,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1734},{"shards":"2,3","objects":2189}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32120,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":273698816,"data_stored":414453,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247843,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673017,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38116,"kb_used_data":7132,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94333724,"statfs":{"total":96636764160,"available":96597733376,"internally_reserved":0,"allocated":7303168,"data_stored":75368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.36799999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38088,"kb_used_data":7104,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94333752,"statfs":{"total":96636764160,"available":96597762048,"internally_reserved":0,"allocated":7274496,"data_stored":75679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.621},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.52900000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.54600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7049216,"data_stored":1721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7094272,"data_stored":1732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:25.962 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:26.274 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:26.274 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":266,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":78032896,"bytes_avail":193195495424,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":32120,"misplaced_total":63330,"misplaced_ratio":0.50718458866256122,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":13,"recovering_keys_per_sec":0,"num_objects_recovered":158,"num_bytes_recovered":158,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.010807995684444904,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.010807995684444904,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:26.291 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:26.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:26.557 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:26.561 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:26.569 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2021-08-21T15:04:25.516666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32120,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":76204,"kb_used_data":14236,"kb_used_omap":0,"kb_used_meta":61952,"kb_avail":188667476,"statfs":{"total":193273528320,"available":193195495424,"internally_reserved":0,"allocated":14577664,"data_stored":151047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63438848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-316,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004193"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9157,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:24.689400+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:24.689400+0000","last_peered":"2021-08-21T15:04:24.689400+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:24.689400+0000","last_undegraded":"2021-08-21T15:04:24.689400+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4378,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1734},{"shards":"2,3","objects":2189}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32120,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":273698816,"data_stored":414453,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247843,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673017,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38116,"kb_used_data":7132,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94333724,"statfs":{"total":96636764160,"available":96597733376,"internally_reserved":0,"allocated":7303168,"data_stored":75368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.36799999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38088,"kb_used_data":7104,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94333752,"statfs":{"total":96636764160,"available":96597762048,"internally_reserved":0,"allocated":7274496,"data_stored":75679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.45000000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.621},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.52900000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.54600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7049216,"data_stored":1721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7094272,"data_stored":1732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:29.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:29.840 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:29.841 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:29.851 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2021-08-21T15:04:29.517849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32120,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":76884,"kb_used_data":14660,"kb_used_omap":0,"kb_used_meta":62208,"kb_avail":188666796,"statfs":{"total":193273528320,"available":193194799104,"internally_reserved":0,"allocated":15011840,"data_stored":151153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63700992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-210,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003943"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9157,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:24.689400+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:24.689400+0000","last_peered":"2021-08-21T15:04:24.689400+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:24.689400+0000","last_undegraded":"2021-08-21T15:04:24.689400+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4378,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1734},{"shards":"2,3","objects":2189}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32120,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274128896,"data_stored":414558,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247843,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673018,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38460,"kb_used_data":7348,"kb_used_omap":0,"kb_used_meta":31104,"kb_avail":94333380,"statfs":{"total":96636764160,"available":96597381120,"internally_reserved":0,"allocated":7524352,"data_stored":75422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31850496},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.499}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.52100000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.55600000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705723,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38424,"kb_used_data":7312,"kb_used_omap":0,"kb_used_meta":31104,"kb_avail":94333416,"statfs":{"total":96636764160,"available":96597417984,"internally_reserved":0,"allocated":7487488,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31850496},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.377},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.46899999999999997}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.495},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7266304,"data_stored":1774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7307264,"data_stored":1784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:29.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:30.156 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":270,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":78729216,"bytes_avail":193194799104,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":32120,"misplaced_total":63330,"misplaced_ratio":0.50718458866256122,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.011148759163916111,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.011148759163916111,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:30.168 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:30.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:30.426 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:30.428 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:30.442 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2021-08-21T15:04:29.517849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32120,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":76884,"kb_used_data":14660,"kb_used_omap":0,"kb_used_meta":62208,"kb_avail":188666796,"statfs":{"total":193273528320,"available":193194799104,"internally_reserved":0,"allocated":15011840,"data_stored":151153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63700992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-210,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003943"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9157,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:24.689400+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:24.689400+0000","last_peered":"2021-08-21T15:04:24.689400+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:24.689400+0000","last_undegraded":"2021-08-21T15:04:24.689400+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4378,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1734},{"shards":"2,3","objects":2189}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32120,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1734,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274128896,"data_stored":414558,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247843,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673018,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38460,"kb_used_data":7348,"kb_used_omap":0,"kb_used_meta":31104,"kb_avail":94333380,"statfs":{"total":96636764160,"available":96597381120,"internally_reserved":0,"allocated":7524352,"data_stored":75422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31850496},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.499}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.52100000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.55600000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705723,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38424,"kb_used_data":7312,"kb_used_omap":0,"kb_used_meta":31104,"kb_avail":94333416,"statfs":{"total":96636764160,"available":96597417984,"internally_reserved":0,"allocated":7487488,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31850496},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.377},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.42999999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.46899999999999997}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.495},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7266304,"data_stored":1774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7307264,"data_stored":1784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:33.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:33.730 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:33.731 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:33.748 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2021-08-21T15:04:33.519029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32014,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1787,"num_bytes_recovered":1787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":77160,"kb_used_data":14872,"kb_used_omap":0,"kb_used_meta":62272,"kb_avail":188666520,"statfs":{"total":193273528320,"available":193194516480,"internally_reserved":0,"allocated":15228928,"data_stored":151206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63766528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003876"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9316,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:29.711076+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:29.711076+0000","last_peered":"2021-08-21T15:04:29.711076+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:29.711076+0000","last_undegraded":"2021-08-21T15:04:29.711076+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4272,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1787,"num_bytes_recovered":1787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1787},{"shards":"2,3","objects":2136}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32014,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1787,"num_bytes_recovered":1787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274345984,"data_stored":414611,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247844,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673018,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38460,"kb_used_data":7348,"kb_used_omap":0,"kb_used_meta":31104,"kb_avail":94333380,"statfs":{"total":96636764160,"available":96597381120,"internally_reserved":0,"allocated":7524352,"data_stored":75422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31850496},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.499}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.52100000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.55600000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38700,"kb_used_data":7524,"kb_used_omap":0,"kb_used_meta":31168,"kb_avail":94333140,"statfs":{"total":96636764160,"available":96597135360,"internally_reserved":0,"allocated":7704576,"data_stored":75784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31916032},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.627}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.58099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7483392,"data_stored":1827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7307264,"data_stored":1784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:33.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:34.059 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:34.059 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":274,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":79011840,"bytes_avail":193194516480,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":32014,"misplaced_total":63330,"misplaced_ratio":0.50551081635875572,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.011148759163916111,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.011148759163916111,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:34.074 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:34.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:34.320 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:34.323 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:34.338 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2021-08-21T15:04:33.519029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32014,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1787,"num_bytes_recovered":1787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":77160,"kb_used_data":14872,"kb_used_omap":0,"kb_used_meta":62272,"kb_avail":188666520,"statfs":{"total":193273528320,"available":193194516480,"internally_reserved":0,"allocated":15228928,"data_stored":151206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63766528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003876"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9316,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:29.711076+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:29.711076+0000","last_peered":"2021-08-21T15:04:29.711076+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:29.711076+0000","last_undegraded":"2021-08-21T15:04:29.711076+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4272,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1787,"num_bytes_recovered":1787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1787},{"shards":"2,3","objects":2136}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":32014,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1787,"num_bytes_recovered":1787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274345984,"data_stored":414611,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247844,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673018,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38460,"kb_used_data":7348,"kb_used_omap":0,"kb_used_meta":31104,"kb_avail":94333380,"statfs":{"total":96636764160,"available":96597381120,"internally_reserved":0,"allocated":7524352,"data_stored":75422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31850496},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.499}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.52100000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.55600000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38700,"kb_used_data":7524,"kb_used_omap":0,"kb_used_meta":31168,"kb_avail":94333140,"statfs":{"total":96636764160,"available":96597135360,"internally_reserved":0,"allocated":7704576,"data_stored":75784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31916032},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.627}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.58099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7483392,"data_stored":1827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7307264,"data_stored":1784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:37.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:37.611 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:37.611 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:37.622 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2021-08-21T15:04:37.519945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31908,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":1840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":77440,"kb_used_data":15088,"kb_used_omap":0,"kb_used_meta":62336,"kb_avail":188666240,"statfs":{"total":193273528320,"available":193194229760,"internally_reserved":0,"allocated":15450112,"data_stored":151260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63832064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003278"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9475,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:34.716994+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:34.716994+0000","last_peered":"2021-08-21T15:04:34.716994+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:34.716994+0000","last_undegraded":"2021-08-21T15:04:34.716994+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4166,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":1840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1840},{"shards":"2,3","objects":2083}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31908,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":1840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274563072,"data_stored":414664,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117026,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673019,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38740,"kb_used_data":7564,"kb_used_omap":0,"kb_used_meta":31168,"kb_avail":94333100,"statfs":{"total":96636764160,"available":96597094400,"internally_reserved":0,"allocated":7745536,"data_stored":75476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31916032},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.499}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.52100000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.55600000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38700,"kb_used_data":7524,"kb_used_omap":0,"kb_used_meta":31168,"kb_avail":94333140,"statfs":{"total":96636764160,"available":96597135360,"internally_reserved":0,"allocated":7704576,"data_stored":75784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31916032},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.627}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.58099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7483392,"data_stored":1827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7524352,"data_stored":1837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:37.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:37.928 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:37.928 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":278,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":79298560,"bytes_avail":193194229760,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":31908,"misplaced_total":63330,"misplaced_ratio":0.50383704405495022,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.011489523574709892,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.011489523574709892,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:37.940 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:37.940 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:38.194 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:38.197 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:38.209 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2021-08-21T15:04:37.519945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31908,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":1840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":77440,"kb_used_data":15088,"kb_used_omap":0,"kb_used_meta":62336,"kb_avail":188666240,"statfs":{"total":193273528320,"available":193194229760,"internally_reserved":0,"allocated":15450112,"data_stored":151260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63832064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003278"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9475,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:34.716994+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:34.716994+0000","last_peered":"2021-08-21T15:04:34.716994+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:34.716994+0000","last_undegraded":"2021-08-21T15:04:34.716994+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4166,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":1840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1840},{"shards":"2,3","objects":2083}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31908,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":1840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274563072,"data_stored":414664,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117026,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673019,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38740,"kb_used_data":7564,"kb_used_omap":0,"kb_used_meta":31168,"kb_avail":94333100,"statfs":{"total":96636764160,"available":96597094400,"internally_reserved":0,"allocated":7745536,"data_stored":75476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31916032},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.499}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.52100000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.55600000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38700,"kb_used_data":7524,"kb_used_omap":0,"kb_used_meta":31168,"kb_avail":94333140,"statfs":{"total":96636764160,"available":96597135360,"internally_reserved":0,"allocated":7704576,"data_stored":75784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31916032},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.627}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.58099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7483392,"data_stored":1827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7524352,"data_stored":1837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:41.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:41.498 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:41.507 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:41.507 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2021-08-21T15:04:39.520589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31908,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":1840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":78000,"kb_used_data":15520,"kb_used_omap":0,"kb_used_meta":62464,"kb_avail":188665680,"statfs":{"total":193273528320,"available":193193656320,"internally_reserved":0,"allocated":15892480,"data_stored":151368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63963136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003391"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9475,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:34.716994+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:34.716994+0000","last_peered":"2021-08-21T15:04:34.716994+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:34.716994+0000","last_undegraded":"2021-08-21T15:04:34.716994+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4166,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":1840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1840},{"shards":"2,3","objects":2083}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31908,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1840,"num_bytes_recovered":1840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274997248,"data_stored":414770,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117026,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673020,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39020,"kb_used_data":7780,"kb_used_omap":0,"kb_used_meta":31232,"kb_avail":94332820,"statfs":{"total":96636764160,"available":96596807680,"internally_reserved":0,"allocated":7966720,"data_stored":75530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31981568},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40600000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.432}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.56499999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705725,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38980,"kb_used_data":7740,"kb_used_omap":0,"kb_used_meta":31232,"kb_avail":94332860,"statfs":{"total":96636764160,"available":96596848640,"internally_reserved":0,"allocated":7925760,"data_stored":75838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31981568},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.5},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.47799999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7700480,"data_stored":1880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7741440,"data_stored":1890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:41.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:41.829 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:41.829 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":282,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":79872000,"bytes_avail":193193656320,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":31908,"misplaced_total":63330,"misplaced_ratio":0.50383704405495022,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.011830287054181099,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.011830287054181099,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:41.845 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:41.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:42.109 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:42.110 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:42.120 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2021-08-21T15:04:41.521227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31802,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1893,"num_bytes_recovered":1893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":78000,"kb_used_data":15520,"kb_used_omap":0,"kb_used_meta":62464,"kb_avail":188665680,"statfs":{"total":193273528320,"available":193193656320,"internally_reserved":0,"allocated":15892480,"data_stored":151368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63963136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-318,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003377"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9634,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:39.724644+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:39.724644+0000","last_peered":"2021-08-21T15:04:39.724644+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:39.724644+0000","last_undegraded":"2021-08-21T15:04:39.724644+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1893,"num_bytes_recovered":1893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1893},{"shards":"2,3","objects":2030}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31802,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1893,"num_bytes_recovered":1893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274997248,"data_stored":414770,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117027,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247846,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673020,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39020,"kb_used_data":7780,"kb_used_omap":0,"kb_used_meta":31232,"kb_avail":94332820,"statfs":{"total":96636764160,"available":96596807680,"internally_reserved":0,"allocated":7966720,"data_stored":75530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31981568},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40600000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.432}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.56499999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705725,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38980,"kb_used_data":7740,"kb_used_omap":0,"kb_used_meta":31232,"kb_avail":94332860,"statfs":{"total":96636764160,"available":96596848640,"internally_reserved":0,"allocated":7925760,"data_stored":75838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31981568},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:03:40 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.5},{"interface":"front","average":{"1min":0.496,"5min":0.499,"15min":0.499},"min":{"1min":0.34300000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.63400000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.47799999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7700480,"data_stored":1880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7741440,"data_stored":1890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:45.124 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:45.382 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:45.384 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:45.400 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2021-08-21T15:04:43.521861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31802,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1893,"num_bytes_recovered":1893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":78336,"kb_used_data":15728,"kb_used_omap":0,"kb_used_meta":62592,"kb_avail":188665344,"statfs":{"total":193273528320,"available":193193312256,"internally_reserved":0,"allocated":16105472,"data_stored":151420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64094208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003327"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9634,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:39.724644+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:39.724644+0000","last_peered":"2021-08-21T15:04:39.724644+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:39.724644+0000","last_undegraded":"2021-08-21T15:04:39.724644+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4060,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1893,"num_bytes_recovered":1893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1893},{"shards":"2,3","objects":2030}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31802,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1893,"num_bytes_recovered":1893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":275214336,"data_stored":414823,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117027,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247846,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673020,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39020,"kb_used_data":7780,"kb_used_omap":0,"kb_used_meta":31232,"kb_avail":94332820,"statfs":{"total":96636764160,"available":96596807680,"internally_reserved":0,"allocated":7966720,"data_stored":75530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31981568},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40600000000000003}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.432}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.56499999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705726,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39316,"kb_used_data":7948,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":94332524,"statfs":{"total":96636764160,"available":96596504576,"internally_reserved":0,"allocated":8138752,"data_stored":75890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.69099999999999995}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.64700000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.71999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7917568,"data_stored":1933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7741440,"data_stored":1890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:45.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:45.707 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:45.707 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":285,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":80216064,"bytes_avail":193193312256,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":31802,"misplaced_total":63330,"misplaced_ratio":0.50216327175114484,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.011830287054181099,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.011830287054181099,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:45.723 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:45.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:45.991 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:45.993 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:46.005 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2021-08-21T15:04:45.522436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31696,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":1946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":78676,"kb_used_data":15940,"kb_used_omap":0,"kb_used_meta":62720,"kb_avail":188665004,"statfs":{"total":193273528320,"available":193192964096,"internally_reserved":0,"allocated":16322560,"data_stored":151473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64225280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-318,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003407"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9793,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:44.727515+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:44.727515+0000","last_peered":"2021-08-21T15:04:44.727515+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:44.727515+0000","last_undegraded":"2021-08-21T15:04:44.727515+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":1946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1946},{"shards":"2,3","objects":1977}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31696,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":1946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":275431424,"data_stored":414876,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673021,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39360,"kb_used_data":7992,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":94332480,"statfs":{"total":96636764160,"available":96596459520,"internally_reserved":0,"allocated":8183808,"data_stored":75583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.63400000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.38900000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.38}]}]},{"osd":0,"up_from":9,"seq":38654705726,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39316,"kb_used_data":7948,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":94332524,"statfs":{"total":96636764160,"available":96596504576,"internally_reserved":0,"allocated":8138752,"data_stored":75890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.69099999999999995}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.64700000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.71999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7917568,"data_stored":1933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7958528,"data_stored":1943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:49.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:49.275 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:49.283 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:49.287 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2021-08-21T15:04:47.522940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31696,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":1946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":78676,"kb_used_data":15940,"kb_used_omap":0,"kb_used_meta":62720,"kb_avail":188665004,"statfs":{"total":193273528320,"available":193192964096,"internally_reserved":0,"allocated":16322560,"data_stored":151473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64225280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003484"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9793,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:44.727515+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:44.727515+0000","last_peered":"2021-08-21T15:04:44.727515+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:44.727515+0000","last_undegraded":"2021-08-21T15:04:44.727515+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":1946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1946},{"shards":"2,3","objects":1977}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31696,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":1946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":275431424,"data_stored":414876,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673021,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39360,"kb_used_data":7992,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":94332480,"statfs":{"total":96636764160,"available":96596459520,"internally_reserved":0,"allocated":8183808,"data_stored":75583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.63400000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:03:44 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44800000000000001,"15min":0.439},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45400000000000001,"15min":0.44800000000000001},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.58999999999999997,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.38900000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.38}]}]},{"osd":0,"up_from":9,"seq":38654705726,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39316,"kb_used_data":7948,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":94332524,"statfs":{"total":96636764160,"available":96596504576,"internally_reserved":0,"allocated":8138752,"data_stored":75890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.69099999999999995}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.64700000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.71999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7917568,"data_stored":1933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7958528,"data_stored":1943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:49.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:49.584 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:49.584 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":289,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":80564224,"bytes_avail":193192964096,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":31696,"misplaced_total":63330,"misplaced_ratio":0.50048949944733934,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.012171050533652306,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.012171050533652306,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:49.597 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:49.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:49.847 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:49.850 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:49.858 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2021-08-21T15:04:49.523648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31696,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":1946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":79228,"kb_used_data":16364,"kb_used_omap":0,"kb_used_meta":62848,"kb_avail":188664452,"statfs":{"total":193273528320,"available":193192398848,"internally_reserved":0,"allocated":16756736,"data_stored":151579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64356352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003703"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9793,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:44.727515+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:44.727515+0000","last_peered":"2021-08-21T15:04:44.727515+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:44.727515+0000","last_undegraded":"2021-08-21T15:04:44.727515+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3954,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":1946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1946},{"shards":"2,3","objects":1977}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31696,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1946,"num_bytes_recovered":1946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":275865600,"data_stored":414982,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673022,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39636,"kb_used_data":8204,"kb_used_omap":0,"kb_used_meta":31424,"kb_avail":94332204,"statfs":{"total":96636764160,"available":96596176896,"internally_reserved":0,"allocated":8400896,"data_stored":75636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32178176},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.38200000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.60999999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.67200000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705727,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39592,"kb_used_data":8160,"kb_used_omap":0,"kb_used_meta":31424,"kb_avail":94332248,"statfs":{"total":96636764160,"available":96596221952,"internally_reserved":0,"allocated":8355840,"data_stored":75943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32178176},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.67000000000000004}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.63},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.60599999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.54600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8134656,"data_stored":1986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8175616,"data_stored":1996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:52.860 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:53.123 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:53.126 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:53.135 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2021-08-21T15:04:51.524225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31590,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1999,"num_bytes_recovered":1999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":79228,"kb_used_data":16364,"kb_used_omap":0,"kb_used_meta":62848,"kb_avail":188664452,"statfs":{"total":193273528320,"available":193192398848,"internally_reserved":0,"allocated":16756736,"data_stored":151579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64356352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-318,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003636"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9952,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:49.729471+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:49.729471+0000","last_peered":"2021-08-21T15:04:49.729471+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:49.729471+0000","last_undegraded":"2021-08-21T15:04:49.729471+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3848,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1999,"num_bytes_recovered":1999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1999},{"shards":"2,3","objects":1924}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31590,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1999,"num_bytes_recovered":1999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":275865600,"data_stored":414982,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117029,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247848,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673022,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39636,"kb_used_data":8204,"kb_used_omap":0,"kb_used_meta":31424,"kb_avail":94332204,"statfs":{"total":96636764160,"available":96596176896,"internally_reserved":0,"allocated":8400896,"data_stored":75636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32178176},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.38200000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.60999999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.67200000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705727,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39592,"kb_used_data":8160,"kb_used_omap":0,"kb_used_meta":31424,"kb_avail":94332248,"statfs":{"total":96636764160,"available":96596221952,"internally_reserved":0,"allocated":8355840,"data_stored":75943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32178176},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.67000000000000004}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.63},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.60599999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.54600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8134656,"data_stored":1986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8175616,"data_stored":1996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:53.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:53.444 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:53.444 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":293,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":81129472,"bytes_avail":193192398848,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":31590,"misplaced_total":63330,"misplaced_ratio":0.49881572714353389,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":13,"recovering_keys_per_sec":0,"num_objects_recovered":159,"num_bytes_recovered":159,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (3m)\n [............................] (remaining: 5h)","progress":0.012511814013123512,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (3m)\n [............................] (remaining: 5h)","progress":0.012511814013123512,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (3m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:53.455 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:53.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:53.715 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:53.716 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:53.724 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2021-08-21T15:04:53.524595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31590,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1999,"num_bytes_recovered":1999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":79568,"kb_used_data":16576,"kb_used_omap":0,"kb_used_meta":62976,"kb_avail":188664112,"statfs":{"total":193273528320,"available":193192050688,"internally_reserved":0,"allocated":16973824,"data_stored":151632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64487424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003368"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":9952,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:49.729471+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:49.729471+0000","last_peered":"2021-08-21T15:04:49.729471+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:49.729471+0000","last_undegraded":"2021-08-21T15:04:49.729471+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3848,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1999,"num_bytes_recovered":1999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1999},{"shards":"2,3","objects":1924}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31590,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1999,"num_bytes_recovered":1999,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276082688,"data_stored":415035,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117029,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247848,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673022,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39636,"kb_used_data":8204,"kb_used_omap":0,"kb_used_meta":31424,"kb_avail":94332204,"statfs":{"total":96636764160,"available":96596176896,"internally_reserved":0,"allocated":8400896,"data_stored":75636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32178176},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.38200000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.60999999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.67200000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39932,"kb_used_data":8372,"kb_used_omap":0,"kb_used_meta":31552,"kb_avail":94331908,"statfs":{"total":96636764160,"available":96595873792,"internally_reserved":0,"allocated":8572928,"data_stored":75996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32309248},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.432},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.50900000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.54300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8351744,"data_stored":2039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8175616,"data_stored":1996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:56.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:56.982 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:56.984 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:56.994 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2021-08-21T15:04:55.525081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31484,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":79844,"kb_used_data":16788,"kb_used_omap":0,"kb_used_meta":63040,"kb_avail":188663836,"statfs":{"total":193273528320,"available":193191768064,"internally_reserved":0,"allocated":17190912,"data_stored":151685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64552960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-318,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003219"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10111,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:54.718120+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:54.718120+0000","last_peered":"2021-08-21T15:04:54.718120+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:54.718120+0000","last_undegraded":"2021-08-21T15:04:54.718120+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3742,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2052},{"shards":"2,3","objects":1871}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31484,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276299776,"data_stored":415088,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117030,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247849,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673023,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39912,"kb_used_data":8416,"kb_used_omap":0,"kb_used_meta":31488,"kb_avail":94331928,"statfs":{"total":96636764160,"available":96595894272,"internally_reserved":0,"allocated":8617984,"data_stored":75689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32243712},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.434}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.46600000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.36399999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39932,"kb_used_data":8372,"kb_used_omap":0,"kb_used_meta":31552,"kb_avail":94331908,"statfs":{"total":96636764160,"available":96595873792,"internally_reserved":0,"allocated":8572928,"data_stored":75996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32309248},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.432},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.50900000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.54300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8351744,"data_stored":2039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8392704,"data_stored":2049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:04:56.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:04:57.305 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:57.305 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":297,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":81760256,"bytes_avail":193191768064,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":31484,"misplaced_total":63330,"misplaced_ratio":0.49714195483972839,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":13,"recovering_keys_per_sec":0,"num_objects_recovered":159,"num_bytes_recovered":159,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 5h)","progress":0.012852577492594719,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 5h)","progress":0.012852577492594719,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (3m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:04:57.319 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:04:57.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:04:57.568 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:04:57.570 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:04:57.581 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2021-08-21T15:04:57.525731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31484,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":79844,"kb_used_data":16788,"kb_used_omap":0,"kb_used_meta":63040,"kb_avail":188663836,"statfs":{"total":193273528320,"available":193191768064,"internally_reserved":0,"allocated":17190912,"data_stored":151685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64552960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003294"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10111,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:54.718120+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:54.718120+0000","last_peered":"2021-08-21T15:04:54.718120+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:54.718120+0000","last_undegraded":"2021-08-21T15:04:54.718120+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3742,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2052},{"shards":"2,3","objects":1871}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31484,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276299776,"data_stored":415088,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117030,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247849,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673023,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39912,"kb_used_data":8416,"kb_used_omap":0,"kb_used_meta":31488,"kb_avail":94331928,"statfs":{"total":96636764160,"available":96595894272,"internally_reserved":0,"allocated":8617984,"data_stored":75689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32243712},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.434}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.46600000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39400000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.36399999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39932,"kb_used_data":8372,"kb_used_omap":0,"kb_used_meta":31552,"kb_avail":94331908,"statfs":{"total":96636764160,"available":96595873792,"internally_reserved":0,"allocated":8572928,"data_stored":75996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32309248},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.432},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.50900000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:03:55 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56799999999999995,"15min":0.57499999999999996},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76400000000000001,"5min":0.88,"15min":0.88},"last":0.54300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8351744,"data_stored":2039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8392704,"data_stored":2049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:00.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:00.849 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:00.851 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:00.860 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2021-08-21T15:04:59.526302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31484,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":80460,"kb_used_data":17212,"kb_used_omap":0,"kb_used_meta":63232,"kb_avail":188663220,"statfs":{"total":193273528320,"available":193191137280,"internally_reserved":0,"allocated":17625088,"data_stored":151791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64749568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003361"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10111,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:54.718120+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:54.718120+0000","last_peered":"2021-08-21T15:04:54.718120+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:54.718120+0000","last_undegraded":"2021-08-21T15:04:54.718120+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3742,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2052},{"shards":"2,3","objects":1871}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31484,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276738048,"data_stored":415195,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117030,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247849,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673024,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40252,"kb_used_data":8628,"kb_used_omap":0,"kb_used_meta":31616,"kb_avail":94331588,"statfs":{"total":96636764160,"available":96595546112,"internally_reserved":0,"allocated":8835072,"data_stored":75742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32374784},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.44400000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46000000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40208,"kb_used_data":8584,"kb_used_omap":0,"kb_used_meta":31616,"kb_avail":94331632,"statfs":{"total":96636764160,"available":96595591168,"internally_reserved":0,"allocated":8790016,"data_stored":76049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32374784},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.378},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.499},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.40500000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.56100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8568832,"data_stored":2092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8613888,"data_stored":2103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:00.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:05:01.166 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:01.167 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":301,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":82391040,"bytes_avail":193191137280,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":31484,"misplaced_total":63330,"misplaced_ratio":0.49714195483972839,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 5h)","progress":0.0131933419033885,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 5h)","progress":0.0131933419033885,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (4m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:05:01.178 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:05:01.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:01.414 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:01.416 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:01.426 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2021-08-21T15:04:59.526302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31484,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":80460,"kb_used_data":17212,"kb_used_omap":0,"kb_used_meta":63232,"kb_avail":188663220,"statfs":{"total":193273528320,"available":193191137280,"internally_reserved":0,"allocated":17625088,"data_stored":151791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64749568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003361"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10111,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:54.718120+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:54.718120+0000","last_peered":"2021-08-21T15:04:54.718120+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:54.718120+0000","last_undegraded":"2021-08-21T15:04:54.718120+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3742,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2052},{"shards":"2,3","objects":1871}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31484,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2052,"num_bytes_recovered":2052,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276738048,"data_stored":415195,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117030,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247849,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673024,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40252,"kb_used_data":8628,"kb_used_omap":0,"kb_used_meta":31616,"kb_avail":94331588,"statfs":{"total":96636764160,"available":96595546112,"internally_reserved":0,"allocated":8835072,"data_stored":75742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32374784},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.44400000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46000000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705729,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40208,"kb_used_data":8584,"kb_used_omap":0,"kb_used_meta":31616,"kb_avail":94331632,"statfs":{"total":96636764160,"available":96595591168,"internally_reserved":0,"allocated":8790016,"data_stored":76049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32374784},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.378},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.499},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.40500000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.56100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8568832,"data_stored":2092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8613888,"data_stored":2103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:04.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:04.704 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:04.706 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:04.715 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2021-08-21T15:05:03.527324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31378,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":2105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":80740,"kb_used_data":17428,"kb_used_omap":0,"kb_used_meta":63296,"kb_avail":188662940,"statfs":{"total":193273528320,"available":193190850560,"internally_reserved":0,"allocated":17846272,"data_stored":151845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64815104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003099"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10270,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:59.684021+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:59.684021+0000","last_peered":"2021-08-21T15:04:59.684021+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:59.684021+0000","last_undegraded":"2021-08-21T15:04:59.684021+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3636,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":2105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2105},{"shards":"2,3","objects":1818}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31378,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":2105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276955136,"data_stored":415248,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247850,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673024,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40252,"kb_used_data":8628,"kb_used_omap":0,"kb_used_meta":31616,"kb_avail":94331588,"statfs":{"total":96636764160,"available":96595546112,"internally_reserved":0,"allocated":8835072,"data_stored":75742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32374784},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.44400000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46000000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40488,"kb_used_data":8800,"kb_used_omap":0,"kb_used_meta":31680,"kb_avail":94331352,"statfs":{"total":96636764160,"available":96595304448,"internally_reserved":0,"allocated":9011200,"data_stored":76103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32440320},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.61299999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.66400000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.54100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8785920,"data_stored":2145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8613888,"data_stored":2103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:04.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:05:05.037 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:05.038 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":305,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":82677760,"bytes_avail":193190850560,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":31378,"misplaced_total":63330,"misplaced_ratio":0.4954681825359229,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 5h)","progress":0.0131933419033885,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 5h)","progress":0.0131933419033885,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (4m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:05:05.048 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:05:05.048 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:05.283 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:05.286 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:05.295 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2021-08-21T15:05:03.527324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31378,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":2105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":80740,"kb_used_data":17428,"kb_used_omap":0,"kb_used_meta":63296,"kb_avail":188662940,"statfs":{"total":193273528320,"available":193190850560,"internally_reserved":0,"allocated":17846272,"data_stored":151845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64815104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003099"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10270,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:04:59.684021+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:04:59.684021+0000","last_peered":"2021-08-21T15:04:59.684021+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:04:59.684021+0000","last_undegraded":"2021-08-21T15:04:59.684021+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3636,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":2105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2105},{"shards":"2,3","objects":1818}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31378,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":2105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276955136,"data_stored":415248,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247850,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673024,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40252,"kb_used_data":8628,"kb_used_omap":0,"kb_used_meta":31616,"kb_avail":94331588,"statfs":{"total":96636764160,"available":96595546112,"internally_reserved":0,"allocated":8835072,"data_stored":75742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32374784},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.35099999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.44400000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46000000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40488,"kb_used_data":8800,"kb_used_omap":0,"kb_used_meta":31680,"kb_avail":94331352,"statfs":{"total":96636764160,"available":96595304448,"internally_reserved":0,"allocated":9011200,"data_stored":76103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32440320},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.61299999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.66400000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.54100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8785920,"data_stored":2145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8613888,"data_stored":2103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:08.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:08.571 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:08.574 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:08.585 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2021-08-21T15:05:07.528852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31270,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2159,"num_bytes_recovered":2159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":81020,"kb_used_data":17644,"kb_used_omap":0,"kb_used_meta":63360,"kb_avail":188662660,"statfs":{"total":193273528320,"available":193190563840,"internally_reserved":0,"allocated":18067456,"data_stored":151899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64880640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-214,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003771"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10432,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:04.740680+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:04.740680+0000","last_peered":"2021-08-21T15:05:04.740680+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:04.740680+0000","last_undegraded":"2021-08-21T15:05:04.740680+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3528,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2159,"num_bytes_recovered":2159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2159},{"shards":"2,3","objects":1764}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31270,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2159,"num_bytes_recovered":2159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":277172224,"data_stored":415301,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117032,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247851,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673025,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40532,"kb_used_data":8844,"kb_used_omap":0,"kb_used_meta":31680,"kb_avail":94331308,"statfs":{"total":96636764160,"available":96595259392,"internally_reserved":0,"allocated":9056256,"data_stored":75796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32440320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.432},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.63}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.38700000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.36499999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40488,"kb_used_data":8800,"kb_used_omap":0,"kb_used_meta":31680,"kb_avail":94331352,"statfs":{"total":96636764160,"available":96595304448,"internally_reserved":0,"allocated":9011200,"data_stored":76103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32440320},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.61299999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.66400000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.54100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8785920,"data_stored":2145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8830976,"data_stored":2156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:08.587 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:05:08.891 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:08.891 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":309,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":82964480,"bytes_avail":193190563840,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":31270,"misplaced_total":63330,"misplaced_ratio":0.4937628296226117,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":107,"num_bytes_recovered":107,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 5h)","progress":0.013534105382859707,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 5h)","progress":0.013534105382859707,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (4m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:05:08.910 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:05:08.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:09.176 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:09.177 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:09.191 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2021-08-21T15:05:07.528852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31270,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2159,"num_bytes_recovered":2159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":81020,"kb_used_data":17644,"kb_used_omap":0,"kb_used_meta":63360,"kb_avail":188662660,"statfs":{"total":193273528320,"available":193190563840,"internally_reserved":0,"allocated":18067456,"data_stored":151899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64880640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-214,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003771"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10432,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:04.740680+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:04.740680+0000","last_peered":"2021-08-21T15:05:04.740680+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:04.740680+0000","last_undegraded":"2021-08-21T15:05:04.740680+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3528,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2159,"num_bytes_recovered":2159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2159},{"shards":"2,3","objects":1764}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31270,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2159,"num_bytes_recovered":2159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":277172224,"data_stored":415301,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117032,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247851,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673025,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40532,"kb_used_data":8844,"kb_used_omap":0,"kb_used_meta":31680,"kb_avail":94331308,"statfs":{"total":96636764160,"available":96595259392,"internally_reserved":0,"allocated":9056256,"data_stored":75796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32440320},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.432},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.63}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.38700000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.36499999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40488,"kb_used_data":8800,"kb_used_omap":0,"kb_used_meta":31680,"kb_avail":94331352,"statfs":{"total":96636764160,"available":96595304448,"internally_reserved":0,"allocated":9011200,"data_stored":76103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32440320},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.61299999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.66400000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.54100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8785920,"data_stored":2145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8830976,"data_stored":2156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:12.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:12.461 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:12.471 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2021-08-21T15:05:11.530099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31164,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2212,"num_bytes_recovered":2212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":81636,"kb_used_data":18068,"kb_used_omap":0,"kb_used_meta":63552,"kb_avail":188662044,"statfs":{"total":193273528320,"available":193189933056,"internally_reserved":0,"allocated":18501632,"data_stored":152005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65077248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-320,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003797"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10591,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:09.718878+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:09.718878+0000","last_peered":"2021-08-21T15:05:09.718878+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:09.718878+0000","last_undegraded":"2021-08-21T15:05:09.718878+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3422,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2212,"num_bytes_recovered":2212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2212},{"shards":"2,3","objects":1711}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31164,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2212,"num_bytes_recovered":2212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":277610496,"data_stored":415408,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673026,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":9052,"kb_used_omap":0,"kb_used_meta":31744,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":9269248,"data_stored":75848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32505856},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.46500000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.38}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.51300000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40832,"kb_used_data":9016,"kb_used_omap":0,"kb_used_meta":31808,"kb_avail":94331008,"statfs":{"total":96636764160,"available":96594952192,"internally_reserved":0,"allocated":9232384,"data_stored":76157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32571392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53200000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.58599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9007104,"data_stored":2199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9048064,"data_stored":2209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:12.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:05:12.781 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:12.782 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":313,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":83595264,"bytes_avail":193189933056,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":31164,"misplaced_total":63330,"misplaced_ratio":0.49208905731880626,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":13,"recovering_keys_per_sec":0,"num_objects_recovered":160,"num_bytes_recovered":160,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 5h)","progress":0.013881298713386059,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 5h)","progress":0.013881298713386059,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (4m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:05:12.797 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:05:12.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:13.048 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:13.052 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:13.060 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2021-08-21T15:05:11.530099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31164,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2212,"num_bytes_recovered":2212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":81636,"kb_used_data":18068,"kb_used_omap":0,"kb_used_meta":63552,"kb_avail":188662044,"statfs":{"total":193273528320,"available":193189933056,"internally_reserved":0,"allocated":18501632,"data_stored":152005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65077248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-320,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003797"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10591,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:09.718878+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:09.718878+0000","last_peered":"2021-08-21T15:05:09.718878+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:09.718878+0000","last_undegraded":"2021-08-21T15:05:09.718878+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3422,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2212,"num_bytes_recovered":2212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2212},{"shards":"2,3","objects":1711}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31164,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2212,"num_bytes_recovered":2212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":277610496,"data_stored":415408,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673026,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40804,"kb_used_data":9052,"kb_used_omap":0,"kb_used_meta":31744,"kb_avail":94331036,"statfs":{"total":96636764160,"available":96594980864,"internally_reserved":0,"allocated":9269248,"data_stored":75848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32505856},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.46500000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.38}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.51300000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40832,"kb_used_data":9016,"kb_used_omap":0,"kb_used_meta":31808,"kb_avail":94331008,"statfs":{"total":96636764160,"available":96594952192,"internally_reserved":0,"allocated":9232384,"data_stored":76157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32571392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.38700000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53200000000000003}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.58599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9007104,"data_stored":2199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9048064,"data_stored":2209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:16.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:16.323 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:16.325 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:16.336 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2021-08-21T15:05:15.531366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31058,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":2265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":82260,"kb_used_data":18500,"kb_used_omap":0,"kb_used_meta":63744,"kb_avail":188661420,"statfs":{"total":193273528320,"available":193189294080,"internally_reserved":0,"allocated":18944000,"data_stored":152113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65273856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-320,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004042"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10750,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:14.677239+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:14.677239+0000","last_peered":"2021-08-21T15:05:14.677239+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:14.677239+0000","last_undegraded":"2021-08-21T15:05:14.677239+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3316,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":2265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2265},{"shards":"2,3","objects":1658}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31058,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":2265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278048768,"data_stored":415515,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673027,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41148,"kb_used_data":9268,"kb_used_omap":0,"kb_used_meta":31872,"kb_avail":94330692,"statfs":{"total":96636764160,"available":96594628608,"internally_reserved":0,"allocated":9490432,"data_stored":75902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32636928},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.28499999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.55100000000000005}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.433}]}]},{"osd":0,"up_from":9,"seq":38654705732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41112,"kb_used_data":9232,"kb_used_omap":0,"kb_used_meta":31872,"kb_avail":94330728,"statfs":{"total":96636764160,"available":96594665472,"internally_reserved":0,"allocated":9453568,"data_stored":76211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32636928},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.30299999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.372},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.40899999999999997}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.57899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9224192,"data_stored":2252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9269248,"data_stored":2263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:16.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:05:16.641 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:16.641 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":316,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":84234240,"bytes_avail":193189294080,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":31058,"misplaced_total":63330,"misplaced_ratio":0.49041528501500081,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":13,"recovering_keys_per_sec":0,"num_objects_recovered":160,"num_bytes_recovered":160,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 5h)","progress":0.014222062192857265,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 5h)","progress":0.014222062192857265,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (4m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:05:16.657 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:05:16.657 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:16.909 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:16.911 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:16.921 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2021-08-21T15:05:15.531366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31058,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":2265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":82260,"kb_used_data":18500,"kb_used_omap":0,"kb_used_meta":63744,"kb_avail":188661420,"statfs":{"total":193273528320,"available":193189294080,"internally_reserved":0,"allocated":18944000,"data_stored":152113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65273856},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-320,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004042"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10750,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:14.677239+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:14.677239+0000","last_peered":"2021-08-21T15:05:14.677239+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:14.677239+0000","last_undegraded":"2021-08-21T15:05:14.677239+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3316,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":2265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2265},{"shards":"2,3","objects":1658}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31058,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":2265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278048768,"data_stored":415515,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673027,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41148,"kb_used_data":9268,"kb_used_omap":0,"kb_used_meta":31872,"kb_avail":94330692,"statfs":{"total":96636764160,"available":96594628608,"internally_reserved":0,"allocated":9490432,"data_stored":75902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32636928},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:04:14 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.47999999999999998,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.68700000000000006,"5min":0.753,"15min":0.753},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.5,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68600000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.28499999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.55100000000000005}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.433}]}]},{"osd":0,"up_from":9,"seq":38654705732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41112,"kb_used_data":9232,"kb_used_omap":0,"kb_used_meta":31872,"kb_avail":94330728,"statfs":{"total":96636764160,"available":96594665472,"internally_reserved":0,"allocated":9453568,"data_stored":76211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32636928},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:04:17 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46200000000000002,"15min":0.48899999999999999},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.504,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.53900000000000003,"5min":1.821,"15min":1.821},"last":0.30299999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.372},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.40899999999999997}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.57899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9224192,"data_stored":2252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9269248,"data_stored":2263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:19.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:20.178 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:20.179 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:20.189 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2021-08-21T15:05:19.532417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31058,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":2265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":82816,"kb_used_data":18928,"kb_used_omap":0,"kb_used_meta":63872,"kb_avail":188660864,"statfs":{"total":193273528320,"available":193188724736,"internally_reserved":0,"allocated":19382272,"data_stored":152220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65404928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003564"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10750,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:14.677239+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:14.677239+0000","last_peered":"2021-08-21T15:05:14.677239+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:14.677239+0000","last_undegraded":"2021-08-21T15:05:14.677239+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3316,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":2265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2265},{"shards":"2,3","objects":1658}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31058,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":2265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278482944,"data_stored":415621,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673028,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41424,"kb_used_data":9480,"kb_used_omap":0,"kb_used_meta":31936,"kb_avail":94330416,"statfs":{"total":96636764160,"available":96594345984,"internally_reserved":0,"allocated":9707520,"data_stored":75955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32702464},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.314},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.50600000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.57099999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.54500000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705733,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41392,"kb_used_data":9448,"kb_used_omap":0,"kb_used_meta":31936,"kb_avail":94330448,"statfs":{"total":96636764160,"available":96594378752,"internally_reserved":0,"allocated":9674752,"data_stored":76265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32702464},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.56999999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.53300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9441280,"data_stored":2305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9486336,"data_stored":2316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:20.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:05:20.480 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:20.480 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":320,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":84803584,"bytes_avail":193188724736,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":31058,"misplaced_total":63330,"misplaced_ratio":0.49041528501500081,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 4h)","progress":0.014562825672328472,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 4h)","progress":0.014562825672328472,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (4m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:05:20.493 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:05:20.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:20.756 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:20.757 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:20.758 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2021-08-21T15:05:19.532417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31058,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":2265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":82816,"kb_used_data":18928,"kb_used_omap":0,"kb_used_meta":63872,"kb_avail":188660864,"statfs":{"total":193273528320,"available":193188724736,"internally_reserved":0,"allocated":19382272,"data_stored":152220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65404928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-212,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":106,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003564"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10750,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:14.677239+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:14.677239+0000","last_peered":"2021-08-21T15:05:14.677239+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:14.677239+0000","last_undegraded":"2021-08-21T15:05:14.677239+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3316,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":2265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2265},{"shards":"2,3","objects":1658}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":31058,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2265,"num_bytes_recovered":2265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278482944,"data_stored":415621,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673028,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41424,"kb_used_data":9480,"kb_used_omap":0,"kb_used_meta":31936,"kb_avail":94330416,"statfs":{"total":96636764160,"available":96594345984,"internally_reserved":0,"allocated":9707520,"data_stored":75955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32702464},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.314},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.50600000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.57099999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.54500000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705733,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41392,"kb_used_data":9448,"kb_used_omap":0,"kb_used_meta":31936,"kb_avail":94330448,"statfs":{"total":96636764160,"available":96594378752,"internally_reserved":0,"allocated":9674752,"data_stored":76265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32702464},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.56999999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.53300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9441280,"data_stored":2305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9486336,"data_stored":2316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:23.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:24.029 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:24.029 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:24.043 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2021-08-21T15:05:23.533453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30950,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2319,"num_bytes_recovered":2319,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":83156,"kb_used_data":19140,"kb_used_omap":0,"kb_used_meta":64000,"kb_avail":188660524,"statfs":{"total":193273528320,"available":193188376576,"internally_reserved":0,"allocated":19599360,"data_stored":152273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65536000},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-214,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003354"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10912,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:19.735312+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:19.735312+0000","last_peered":"2021-08-21T15:05:19.735312+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:19.735312+0000","last_undegraded":"2021-08-21T15:05:19.735312+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3208,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2319,"num_bytes_recovered":2319,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2319},{"shards":"2,3","objects":1604}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30950,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2319,"num_bytes_recovered":2319,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278704128,"data_stored":415675,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117035,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247854,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673028,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41424,"kb_used_data":9480,"kb_used_omap":0,"kb_used_meta":31936,"kb_avail":94330416,"statfs":{"total":96636764160,"available":96594345984,"internally_reserved":0,"allocated":9707520,"data_stored":75955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32702464},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.314},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.50600000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.57099999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.54500000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41732,"kb_used_data":9660,"kb_used_omap":0,"kb_used_meta":32064,"kb_avail":94330108,"statfs":{"total":96636764160,"available":96594030592,"internally_reserved":0,"allocated":9891840,"data_stored":76318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32833536},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.49199999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.53300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9662464,"data_stored":2359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9486336,"data_stored":2316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:24.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:05:24.355 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:24.355 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":324,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":85151744,"bytes_avail":193188376576,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":30950,"misplaced_total":63330,"misplaced_ratio":0.48870993210168956,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":107,"num_bytes_recovered":107,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 4h)","progress":0.014562825672328472,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 4h)","progress":0.014562825672328472,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (4m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:05:24.366 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:05:24.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:24.624 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:24.625 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:24.648 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2021-08-21T15:05:23.533453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30950,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2319,"num_bytes_recovered":2319,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":83156,"kb_used_data":19140,"kb_used_omap":0,"kb_used_meta":64000,"kb_avail":188660524,"statfs":{"total":193273528320,"available":193188376576,"internally_reserved":0,"allocated":19599360,"data_stored":152273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65536000},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-214,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003354"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":10912,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:19.735312+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:19.735312+0000","last_peered":"2021-08-21T15:05:19.735312+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:19.735312+0000","last_undegraded":"2021-08-21T15:05:19.735312+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3208,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2319,"num_bytes_recovered":2319,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2319},{"shards":"2,3","objects":1604}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30950,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2319,"num_bytes_recovered":2319,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278704128,"data_stored":415675,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117035,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247854,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673028,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41424,"kb_used_data":9480,"kb_used_omap":0,"kb_used_meta":31936,"kb_avail":94330416,"statfs":{"total":96636764160,"available":96594345984,"internally_reserved":0,"allocated":9707520,"data_stored":75955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32702464},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.314},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.50600000000000001}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.57099999999999995}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.54500000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41732,"kb_used_data":9660,"kb_used_omap":0,"kb_used_meta":32064,"kb_avail":94330108,"statfs":{"total":96636764160,"available":96594030592,"internally_reserved":0,"allocated":9891840,"data_stored":76318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32833536},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.49199999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.53300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9662464,"data_stored":2359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9486336,"data_stored":2316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:27.651 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:27.904 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:27.906 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:27.915 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2021-08-21T15:05:27.534696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30844,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2372,"num_bytes_recovered":2372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":83432,"kb_used_data":19352,"kb_used_omap":0,"kb_used_meta":64064,"kb_avail":188660248,"statfs":{"total":193273528320,"available":193188093952,"internally_reserved":0,"allocated":19816448,"data_stored":152326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65601536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-214,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003330"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":11071,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:24.688004+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:24.688004+0000","last_peered":"2021-08-21T15:05:24.688004+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:24.688004+0000","last_undegraded":"2021-08-21T15:05:24.688004+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3102,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2372,"num_bytes_recovered":2372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2372},{"shards":"2,3","objects":1551}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30844,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2372,"num_bytes_recovered":2372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278925312,"data_stored":415729,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117036,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247855,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673029,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41700,"kb_used_data":9692,"kb_used_omap":0,"kb_used_meta":32000,"kb_avail":94330140,"statfs":{"total":96636764160,"available":96594063360,"internally_reserved":0,"allocated":9924608,"data_stored":76008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32768000},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.313},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.52500000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.66400000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41732,"kb_used_data":9660,"kb_used_omap":0,"kb_used_meta":32064,"kb_avail":94330108,"statfs":{"total":96636764160,"available":96594030592,"internally_reserved":0,"allocated":9891840,"data_stored":76318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32833536},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.49199999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.53300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9662464,"data_stored":2359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9707520,"data_stored":2370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:27.916 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:05:28.235 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:28.235 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":328,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":85434368,"bytes_avail":193188093952,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":30844,"misplaced_total":63330,"misplaced_ratio":0.48703615979788412,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":107,"num_bytes_recovered":107,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 4h)","progress":0.014910019002854824,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 4h)","progress":0.014910019002854824,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (4m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:05:28.247 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:05:28.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:28.500 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:28.501 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:28.512 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2021-08-21T15:05:27.534696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30844,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2372,"num_bytes_recovered":2372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":83432,"kb_used_data":19352,"kb_used_omap":0,"kb_used_meta":64064,"kb_avail":188660248,"statfs":{"total":193273528320,"available":193188093952,"internally_reserved":0,"allocated":19816448,"data_stored":152326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65601536},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-214,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003330"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":11071,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:24.688004+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:24.688004+0000","last_peered":"2021-08-21T15:05:24.688004+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:24.688004+0000","last_undegraded":"2021-08-21T15:05:24.688004+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3102,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2372,"num_bytes_recovered":2372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2372},{"shards":"2,3","objects":1551}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30844,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2372,"num_bytes_recovered":2372,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278925312,"data_stored":415729,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117036,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247855,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673029,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41700,"kb_used_data":9692,"kb_used_omap":0,"kb_used_meta":32000,"kb_avail":94330140,"statfs":{"total":96636764160,"available":96594063360,"internally_reserved":0,"allocated":9924608,"data_stored":76008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32768000},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.313},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.41299999999999998}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.52500000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.66400000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41732,"kb_used_data":9660,"kb_used_omap":0,"kb_used_meta":32064,"kb_avail":94330108,"statfs":{"total":96636764160,"available":96594030592,"internally_reserved":0,"allocated":9891840,"data_stored":76318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32833536},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.49199999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.53300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9662464,"data_stored":2359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9707520,"data_stored":2370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:31.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:31.772 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:31.773 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:31.786 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2021-08-21T15:05:31.535833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30736,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2426,"num_bytes_recovered":2426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":84056,"kb_used_data":19784,"kb_used_omap":0,"kb_used_meta":64256,"kb_avail":188659624,"statfs":{"total":193273528320,"available":193187454976,"internally_reserved":0,"allocated":20258816,"data_stored":152434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65798144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-322,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":161,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003416"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":11233,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:29.728917+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:29.728917+0000","last_peered":"2021-08-21T15:05:29.728917+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:29.728917+0000","last_undegraded":"2021-08-21T15:05:29.728917+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2994,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2426,"num_bytes_recovered":2426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2426},{"shards":"2,3","objects":1497}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30736,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2426,"num_bytes_recovered":2426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":279359488,"data_stored":415835,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117037,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247856,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673030,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42044,"kb_used_data":9908,"kb_used_omap":0,"kb_used_meta":32128,"kb_avail":94329796,"statfs":{"total":96636764160,"available":96593711104,"internally_reserved":0,"allocated":10145792,"data_stored":76062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32899072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.497}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.46000000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.54000000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705735,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42012,"kb_used_data":9876,"kb_used_omap":0,"kb_used_meta":32128,"kb_avail":94329828,"statfs":{"total":96636764160,"available":96593743872,"internally_reserved":0,"allocated":10113024,"data_stored":76372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32899072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.498}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.56299999999999994}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9879552,"data_stored":2412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9924608,"data_stored":2423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:31.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:05:32.093 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:32.093 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":332,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":86073344,"bytes_avail":193187454976,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":30736,"misplaced_total":63330,"misplaced_ratio":0.48533080688457292,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":13,"recovering_keys_per_sec":0,"num_objects_recovered":161,"num_bytes_recovered":161,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 4h)","progress":0.015250782482326031,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 4h)","progress":0.015250782482326031,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (4m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:05:32.105 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:05:32.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:32.369 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:32.371 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:32.380 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2021-08-21T15:05:31.535833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30736,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2426,"num_bytes_recovered":2426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":84056,"kb_used_data":19784,"kb_used_omap":0,"kb_used_meta":64256,"kb_avail":188659624,"statfs":{"total":193273528320,"available":193187454976,"internally_reserved":0,"allocated":20258816,"data_stored":152434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65798144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-322,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":161,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003416"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":11233,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:29.728917+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:29.728917+0000","last_peered":"2021-08-21T15:05:29.728917+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:29.728917+0000","last_undegraded":"2021-08-21T15:05:29.728917+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2994,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2426,"num_bytes_recovered":2426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2426},{"shards":"2,3","objects":1497}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30736,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2426,"num_bytes_recovered":2426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":279359488,"data_stored":415835,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117037,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247856,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673030,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42044,"kb_used_data":9908,"kb_used_omap":0,"kb_used_meta":32128,"kb_avail":94329796,"statfs":{"total":96636764160,"available":96593711104,"internally_reserved":0,"allocated":10145792,"data_stored":76062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32899072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.497}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.46000000000000002}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.54000000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705735,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42012,"kb_used_data":9876,"kb_used_omap":0,"kb_used_meta":32128,"kb_avail":94329828,"statfs":{"total":96636764160,"available":96593743872,"internally_reserved":0,"allocated":10113024,"data_stored":76372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32899072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.498}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.56299999999999994}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9879552,"data_stored":2412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9924608,"data_stored":2423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:35.382 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:35.636 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:35.638 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:35.647 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2021-08-21T15:05:35.537024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30628,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":2480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":84616,"kb_used_data":20216,"kb_used_omap":0,"kb_used_meta":64384,"kb_avail":188659064,"statfs":{"total":193273528320,"available":193186881536,"internally_reserved":0,"allocated":20701184,"data_stored":152542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65929216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-322,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":161,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003570"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":11395,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:34.756048+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:34.756048+0000","last_peered":"2021-08-21T15:05:34.756048+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:34.756048+0000","last_undegraded":"2021-08-21T15:05:34.756048+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2886,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":2480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2480},{"shards":"2,3","objects":1443}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30628,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":2480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":279801856,"data_stored":415943,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247857,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673031,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42320,"kb_used_data":10120,"kb_used_omap":0,"kb_used_meta":32192,"kb_avail":94329520,"statfs":{"total":96636764160,"available":96593428480,"internally_reserved":0,"allocated":10362880,"data_stored":76115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32964608},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.377},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.42499999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.59499999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42296,"kb_used_data":10096,"kb_used_omap":0,"kb_used_meta":32192,"kb_avail":94329544,"statfs":{"total":96636764160,"available":96593453056,"internally_reserved":0,"allocated":10338304,"data_stored":76427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32964608},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.54500000000000004}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10100736,"data_stored":2466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10145792,"data_stored":2477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:35.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:05:35.955 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:35.955 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":336,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":86364160,"bytes_avail":193187164160,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":30736,"misplaced_total":63330,"misplaced_ratio":0.48533080688457292,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":107,"num_bytes_recovered":107,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 4h)","progress":0.015250782482326031,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 4h)","progress":0.015250782482326031,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (4m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:05:35.972 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:05:35.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:36.245 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:36.249 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:36.261 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2021-08-21T15:05:35.537024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30628,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":2480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":84616,"kb_used_data":20216,"kb_used_omap":0,"kb_used_meta":64384,"kb_avail":188659064,"statfs":{"total":193273528320,"available":193186881536,"internally_reserved":0,"allocated":20701184,"data_stored":152542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65929216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-322,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":161,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003570"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":11395,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:34.756048+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:34.756048+0000","last_peered":"2021-08-21T15:05:34.756048+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:34.756048+0000","last_undegraded":"2021-08-21T15:05:34.756048+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2886,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":2480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2480},{"shards":"2,3","objects":1443}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30628,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":2480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":279801856,"data_stored":415943,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247857,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673031,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42320,"kb_used_data":10120,"kb_used_omap":0,"kb_used_meta":32192,"kb_avail":94329520,"statfs":{"total":96636764160,"available":96593428480,"internally_reserved":0,"allocated":10362880,"data_stored":76115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32964608},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.377},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.42499999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.59499999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42296,"kb_used_data":10096,"kb_used_omap":0,"kb_used_meta":32192,"kb_avail":94329544,"statfs":{"total":96636764160,"available":96593453056,"internally_reserved":0,"allocated":10338304,"data_stored":76427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32964608},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.54500000000000004}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10100736,"data_stored":2466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10145792,"data_stored":2477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:39.264 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:39.503 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:39.505 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:39.513 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2021-08-21T15:05:37.537547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30628,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":2480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":84616,"kb_used_data":20216,"kb_used_omap":0,"kb_used_meta":64384,"kb_avail":188659064,"statfs":{"total":193273528320,"available":193186881536,"internally_reserved":0,"allocated":20701184,"data_stored":152542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":65929216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-216,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":108,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003507"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":11395,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:34.756048+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:34.756048+0000","last_peered":"2021-08-21T15:05:34.756048+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:34.756048+0000","last_undegraded":"2021-08-21T15:05:34.756048+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2886,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":2480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2480},{"shards":"2,3","objects":1443}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30628,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":2480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":279801856,"data_stored":415943,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247857,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673031,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42320,"kb_used_data":10120,"kb_used_omap":0,"kb_used_meta":32192,"kb_avail":94329520,"statfs":{"total":96636764160,"available":96593428480,"internally_reserved":0,"allocated":10362880,"data_stored":76115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32964608},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.377},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.42499999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.59499999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42296,"kb_used_data":10096,"kb_used_omap":0,"kb_used_meta":32192,"kb_avail":94329544,"statfs":{"total":96636764160,"available":96593453056,"internally_reserved":0,"allocated":10338304,"data_stored":76427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32964608},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.35899999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.54500000000000004}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10100736,"data_stored":2466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10145792,"data_stored":2477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:39.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:05:39.817 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:39.817 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":340,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":86646784,"bytes_avail":193186881536,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":30628,"misplaced_total":63330,"misplaced_ratio":0.48362545397126167,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":108,"num_bytes_recovered":108,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 4h)","progress":0.015597975812852383,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 4h)","progress":0.015597975812852383,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (4m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:05:39.836 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:05:39.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:40.089 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:40.091 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:40.100 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2021-08-21T15:05:39.538104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30628,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":2480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":85232,"kb_used_data":20640,"kb_used_omap":0,"kb_used_meta":64576,"kb_avail":188658448,"statfs":{"total":193273528320,"available":193186250752,"internally_reserved":0,"allocated":21135360,"data_stored":152648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66125824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-216,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":108,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003407"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":11395,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:34.756048+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:34.756048+0000","last_peered":"2021-08-21T15:05:34.756048+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:34.756048+0000","last_undegraded":"2021-08-21T15:05:34.756048+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2886,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":2480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2480},{"shards":"2,3","objects":1443}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30628,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":2480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":280244224,"data_stored":416051,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247857,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673032,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42596,"kb_used_data":10332,"kb_used_omap":0,"kb_used_meta":32256,"kb_avail":94329244,"statfs":{"total":96636764160,"available":96593145856,"internally_reserved":0,"allocated":10579968,"data_stored":76168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33030144},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.55800000000000005}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.495},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.51900000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705737,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42636,"kb_used_data":10308,"kb_used_omap":0,"kb_used_meta":32320,"kb_avail":94329204,"statfs":{"total":96636764160,"available":96593104896,"internally_reserved":0,"allocated":10555392,"data_stored":76480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33095680},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.48999999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.29899999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10321920,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10366976,"data_stored":2531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:43.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:43.431 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:43.432 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:43.433 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2021-08-21T15:05:41.538558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30522,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2533,"num_bytes_recovered":2533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":85232,"kb_used_data":20640,"kb_used_omap":0,"kb_used_meta":64576,"kb_avail":188658448,"statfs":{"total":193273528320,"available":193186250752,"internally_reserved":0,"allocated":21135360,"data_stored":152648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66125824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-322,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":161,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003329"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":11554,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:39.691295+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:39.691295+0000","last_peered":"2021-08-21T15:05:39.691295+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:39.691295+0000","last_undegraded":"2021-08-21T15:05:39.691295+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2780,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2533,"num_bytes_recovered":2533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2533},{"shards":"2,3","objects":1390}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30522,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2533,"num_bytes_recovered":2533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":280244224,"data_stored":416051,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247858,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673032,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42596,"kb_used_data":10332,"kb_used_omap":0,"kb_used_meta":32256,"kb_avail":94329244,"statfs":{"total":96636764160,"available":96593145856,"internally_reserved":0,"allocated":10579968,"data_stored":76168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33030144},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.55800000000000005}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.495},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.51900000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705737,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42636,"kb_used_data":10308,"kb_used_omap":0,"kb_used_meta":32320,"kb_avail":94329204,"statfs":{"total":96636764160,"available":96593104896,"internally_reserved":0,"allocated":10555392,"data_stored":76480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33095680},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.48999999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.29899999999999999}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10321920,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10366976,"data_stored":2531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:43.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:05:43.759 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:43.759 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":344,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":87277568,"bytes_avail":193186250752,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":30522,"misplaced_total":63330,"misplaced_ratio":0.48195168166745617,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":13,"recovering_keys_per_sec":0,"num_objects_recovered":161,"num_bytes_recovered":161,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 4h)","progress":0.015945170074701309,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 4h)","progress":0.015945170074701309,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (4m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:05:43.772 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:05:43.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:44.031 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:44.032 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:44.043 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2021-08-21T15:05:43.539186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30522,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2533,"num_bytes_recovered":2533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":85512,"kb_used_data":20856,"kb_used_omap":0,"kb_used_meta":64640,"kb_avail":188658168,"statfs":{"total":193273528320,"available":193185964032,"internally_reserved":0,"allocated":21356544,"data_stored":152702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66191360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-214,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003352"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":11554,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:39.691295+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:39.691295+0000","last_peered":"2021-08-21T15:05:39.691295+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:39.691295+0000","last_undegraded":"2021-08-21T15:05:39.691295+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2780,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2533,"num_bytes_recovered":2533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2533},{"shards":"2,3","objects":1390}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30522,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2533,"num_bytes_recovered":2533,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":280465408,"data_stored":416105,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247858,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673032,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42596,"kb_used_data":10332,"kb_used_omap":0,"kb_used_meta":32256,"kb_avail":94329244,"statfs":{"total":96636764160,"available":96593145856,"internally_reserved":0,"allocated":10579968,"data_stored":76168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33030144},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.34000000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.55800000000000005}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.495},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.51900000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42916,"kb_used_data":10524,"kb_used_omap":0,"kb_used_meta":32384,"kb_avail":94328924,"statfs":{"total":96636764160,"available":96592818176,"internally_reserved":0,"allocated":10776576,"data_stored":76534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33161216},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.42799999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.52000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10543104,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10366976,"data_stored":2531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:47.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:47.320 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:47.323 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:47.342 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2021-08-21T15:05:45.539658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30414,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2587,"num_bytes_recovered":2587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":85852,"kb_used_data":21068,"kb_used_omap":0,"kb_used_meta":64768,"kb_avail":188657828,"statfs":{"total":193273528320,"available":193185615872,"internally_reserved":0,"allocated":21573632,"data_stored":152755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66322432},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-322,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":161,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003384"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":11716,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:44.721205+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:44.721205+0000","last_peered":"2021-08-21T15:05:44.721205+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:44.721205+0000","last_undegraded":"2021-08-21T15:05:44.721205+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2672,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2587,"num_bytes_recovered":2587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2587},{"shards":"2,3","objects":1336}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30414,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2587,"num_bytes_recovered":2587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":280682496,"data_stored":416158,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247859,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673033,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42936,"kb_used_data":10544,"kb_used_omap":0,"kb_used_meta":32384,"kb_avail":94328904,"statfs":{"total":96636764160,"available":96592797696,"internally_reserved":0,"allocated":10797056,"data_stored":76221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33161216},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.501}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.63200000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46400000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42916,"kb_used_data":10524,"kb_used_omap":0,"kb_used_meta":32384,"kb_avail":94328924,"statfs":{"total":96636764160,"available":96592818176,"internally_reserved":0,"allocated":10776576,"data_stored":76534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33161216},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.42799999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.52000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10543104,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10584064,"data_stored":2584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:47.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:05:47.677 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:47.677 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":347,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":87912448,"bytes_avail":193185615872,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":30414,"misplaced_total":63330,"misplaced_ratio":0.48024632875414497,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":13,"recovering_keys_per_sec":0,"num_objects_recovered":161,"num_bytes_recovered":161,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 4h)","progress":0.016285933554172516,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 4h)","progress":0.016285933554172516,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (4m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:05:47.695 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:05:47.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:47.956 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:47.959 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:47.969 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2021-08-21T15:05:47.540373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30414,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2587,"num_bytes_recovered":2587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":85852,"kb_used_data":21068,"kb_used_omap":0,"kb_used_meta":64768,"kb_avail":188657828,"statfs":{"total":193273528320,"available":193185615872,"internally_reserved":0,"allocated":21573632,"data_stored":152755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66322432},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-214,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003349"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":11716,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:44.721205+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:44.721205+0000","last_peered":"2021-08-21T15:05:44.721205+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:44.721205+0000","last_undegraded":"2021-08-21T15:05:44.721205+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2672,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2587,"num_bytes_recovered":2587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2587},{"shards":"2,3","objects":1336}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30414,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2587,"num_bytes_recovered":2587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":280682496,"data_stored":416158,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247859,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673033,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42936,"kb_used_data":10544,"kb_used_omap":0,"kb_used_meta":32384,"kb_avail":94328904,"statfs":{"total":96636764160,"available":96592797696,"internally_reserved":0,"allocated":10797056,"data_stored":76221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33161216},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.501}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.63200000000000001}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46400000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42916,"kb_used_data":10524,"kb_used_omap":0,"kb_used_meta":32384,"kb_avail":94328924,"statfs":{"total":96636764160,"available":96592818176,"internally_reserved":0,"allocated":10776576,"data_stored":76534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33161216},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.42799999999999999}]},{"osd":2,"last update":"Sat Aug 21 15:04:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63400000000000001,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.498},"min":{"1min":0.255,"5min":0.251,"15min":0.251},"max":{"1min":0.64700000000000002,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.52000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10543104,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10584064,"data_stored":2584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:50.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:51.232 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:51.234 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:51.247 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2021-08-21T15:05:49.540757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30414,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2587,"num_bytes_recovered":2587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":86476,"kb_used_data":21500,"kb_used_omap":0,"kb_used_meta":64960,"kb_avail":188657204,"statfs":{"total":193273528320,"available":193184976896,"internally_reserved":0,"allocated":22016000,"data_stored":152863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66519040},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-214,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003209"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":11716,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:44.721205+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:44.721205+0000","last_peered":"2021-08-21T15:05:44.721205+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:44.721205+0000","last_undegraded":"2021-08-21T15:05:44.721205+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2672,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2587,"num_bytes_recovered":2587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2587},{"shards":"2,3","objects":1336}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30414,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2587,"num_bytes_recovered":2587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":281120768,"data_stored":416265,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247859,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673034,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43220,"kb_used_data":10764,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94328620,"statfs":{"total":96636764160,"available":96592506880,"internally_reserved":0,"allocated":11022336,"data_stored":76276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.441},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.33500000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.311}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.50800000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705739,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43256,"kb_used_data":10736,"kb_used_omap":0,"kb_used_meta":32512,"kb_avail":94328584,"statfs":{"total":96636764160,"available":96592470016,"internally_reserved":0,"allocated":10993664,"data_stored":76587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33292288},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Aug 21 15:05:45 2021","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.30399999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.496,"15min":0.498},"min":{"1min":0.29899999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.66600000000000004,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.41599999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10760192,"data_stored":2627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10805248,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:51.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2021-08-21T15:05:51.577 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:51.577 INFO:teuthology.orchestra.run.smithi165.stdout:{"fsid":"b82e8492-200f-4d81-953f-d1ffd38e4baa","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27742/63330 objects degraded (43.805%), 7 pgs degraded, 8 pgs undersized","count":15},"muted":false}},"mutes":[]},"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":351,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":29,"num_osds":4,"num_up_osds":4,"osd_up_since":1629558110,"num_in_osds":2,"osd_in_since":1629558052,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":31665,"data_bytes":205500,"bytes_used":88551424,"bytes_avail":193184976896,"bytes_total":193273528320,"degraded_objects":27742,"degraded_total":63330,"degraded_ratio":0.43805463445444492,"misplaced_objects":30414,"misplaced_total":63330,"misplaced_ratio":0.48024632875414497,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8,"recovering_keys_per_sec":0,"num_objects_recovered":107,"num_bytes_recovered":107,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2021-08-21T15:02:51.488283+0000","services":{}},"progress_events":{"1d845a39-f91f-4700-a0e8-2c1b75ec5d3c":{"message":"Rebalancing after osd.0 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false},"5ec8e6e9-ddf9-4800-be89-db0fd109615b":{"message":"Rebalancing after osd.2 marked out (4m)\n [............................] (remaining: 4h)","progress":0.016633126884698868,"add_to_ceph_s":false},"b76d601d-b890-4d7a-9557-b6361aca0b52":{"message":"Rebalancing after osd.3 marked out (4m)\n [............................] (remaining: 4h)","progress":0.016633126884698868,"add_to_ceph_s":false},"e6fb09c2-bb5d-4094-b0ef-938c511d27c2":{"message":"Global Recovery Event (4m)\n [............................] ","progress":0,"add_to_ceph_s":true},"ff8c034e-9d62-4aeb-bb21-acf47dc5d977":{"message":"Rebalancing after osd.1 marked in (4m)\n [............................] ","progress":0,"add_to_ceph_s":false}}} 2021-08-21T15:05:51.590 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2021-08-21T15:05:51.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-08-21T15:05:51.857 INFO:teuthology.orchestra.run.smithi165.stdout: 2021-08-21T15:05:51.860 INFO:teuthology.orchestra.run.smithi165.stderr:dumped all 2021-08-21T15:05:51.871 INFO:teuthology.orchestra.run.smithi165.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2021-08-21T15:05:51.541126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30306,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2641,"num_bytes_recovered":2641,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":86476,"kb_used_data":21500,"kb_used_omap":0,"kb_used_meta":64960,"kb_avail":188657204,"statfs":{"total":193273528320,"available":193184976896,"internally_reserved":0,"allocated":22016000,"data_stored":152863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":66519040},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-322,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":161,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003021"},"pg_stats":[{"pgid":"1.7","version":"15'3956","reported_seq":3988,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885975+0000","last_change":"2021-08-21T15:01:37.913014+0000","last_active":"2021-08-21T15:01:51.885975+0000","last_peered":"2021-08-21T15:01:51.885975+0000","last_clean":"2021-08-21T15:00:48.699483+0000","last_became_active":"2021-08-21T15:01:37.904536+0000","last_became_peered":"2021-08-21T15:01:37.904536+0000","last_unstale":"2021-08-21T15:01:51.885975+0000","last_undegraded":"2021-08-21T15:01:37.895567+0000","last_fullsized":"2021-08-21T15:01:37.780695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3956,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3956,"num_objects":3956,"num_object_clones":0,"num_object_copies":7912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3956,"num_objects_misplaced":3956,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":3956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3956}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"15'3896","reported_seq":3936,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.998766+0000","last_change":"2021-08-21T15:01:53.033876+0000","last_active":"2021-08-21T15:01:52.998766+0000","last_peered":"2021-08-21T15:01:52.998766+0000","last_clean":"2021-08-21T15:00:50.527252+0000","last_became_active":"2021-08-21T15:01:52.994499+0000","last_became_peered":"2021-08-21T15:01:52.994499+0000","last_unstale":"2021-08-21T15:01:52.998766+0000","last_undegraded":"2021-08-21T15:01:52.988052+0000","last_fullsized":"2021-08-21T15:01:52.890213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3896,"ondisk_log_size":3896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40488,"num_objects":3896,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3896,"num_objects_misplaced":3896,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":3928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3896}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"15'3859","reported_seq":3893,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885086+0000","last_change":"2021-08-21T15:01:37.918872+0000","last_active":"2021-08-21T15:01:51.885086+0000","last_peered":"2021-08-21T15:01:51.885086+0000","last_clean":"2021-08-21T15:00:52.399475+0000","last_became_active":"2021-08-21T15:01:37.913326+0000","last_became_peered":"2021-08-21T15:01:37.913326+0000","last_unstale":"2021-08-21T15:01:51.885086+0000","last_undegraded":"2021-08-21T15:01:37.903601+0000","last_fullsized":"2021-08-21T15:01:37.776341+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3859,"ondisk_log_size":3859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22178,"num_objects":3859,"num_object_clones":0,"num_object_copies":7718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3859,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3859,"num_write_kb":3875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3859}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"15'3986","reported_seq":4026,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:53.032177+0000","last_change":"2021-08-21T15:01:53.074185+0000","last_active":"2021-08-21T15:01:53.032177+0000","last_peered":"2021-08-21T15:01:53.032177+0000","last_clean":"2021-08-21T15:00:50.800815+0000","last_became_active":"2021-08-21T15:01:53.024748+0000","last_became_peered":"2021-08-21T15:01:53.024748+0000","last_unstale":"2021-08-21T15:01:53.032177+0000","last_undegraded":"2021-08-21T15:01:53.015856+0000","last_fullsized":"2021-08-21T15:01:52.889903+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3986,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40578,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3986,"num_objects_misplaced":3986,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":4018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3986}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"15'3988","reported_seq":4022,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.886437+0000","last_change":"2021-08-21T15:01:37.931383+0000","last_active":"2021-08-21T15:01:51.886437+0000","last_peered":"2021-08-21T15:01:51.886437+0000","last_clean":"2021-08-21T15:00:50.372465+0000","last_became_active":"2021-08-21T15:01:37.926889+0000","last_became_peered":"2021-08-21T15:01:37.926889+0000","last_unstale":"2021-08-21T15:01:51.886437+0000","last_undegraded":"2021-08-21T15:01:37.920893+0000","last_fullsized":"2021-08-21T15:01:37.784190+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3988,"ondisk_log_size":3988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22284,"num_objects":3988,"num_object_clones":0,"num_object_copies":7976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3988,"num_objects_misplaced":3988,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":4004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":3988}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'3923","reported_seq":11878,"reported_epoch":29,"state":"active+recovering+undersized+remapped","last_fresh":"2021-08-21T15:05:49.727081+0000","last_change":"2021-08-21T15:01:51.024599+0000","last_active":"2021-08-21T15:05:49.727081+0000","last_peered":"2021-08-21T15:05:49.727081+0000","last_clean":"2021-08-21T15:00:50.696737+0000","last_became_active":"2021-08-21T15:01:37.888763+0000","last_became_peered":"2021-08-21T15:01:37.888763+0000","last_unstale":"2021-08-21T15:05:49.727081+0000","last_undegraded":"2021-08-21T15:05:49.727081+0000","last_fullsized":"2021-08-21T15:01:37.776768+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":3923,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22219,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2564,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":3939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2641,"num_bytes_recovered":2641,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2641},{"shards":"2,3","objects":1282}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"15'4036","reported_seq":4076,"reported_epoch":29,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:52.966063+0000","last_change":"2021-08-21T15:01:53.000491+0000","last_active":"2021-08-21T15:01:52.966063+0000","last_peered":"2021-08-21T15:01:52.966063+0000","last_clean":"2021-08-21T15:00:50.216409+0000","last_became_active":"2021-08-21T15:01:52.961995+0000","last_became_peered":"2021-08-21T15:01:52.961995+0000","last_unstale":"2021-08-21T15:01:52.966063+0000","last_undegraded":"2021-08-21T15:01:52.954201+0000","last_fullsized":"2021-08-21T15:01:52.888481+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4036,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13184,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4036,"num_objects_misplaced":4036,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":4044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4036}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'4021","reported_seq":4053,"reported_epoch":28,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-08-21T15:01:51.885048+0000","last_change":"2021-08-21T15:01:37.900378+0000","last_active":"2021-08-21T15:01:51.885048+0000","last_peered":"2021-08-21T15:01:51.885048+0000","last_clean":"2021-08-21T15:00:50.856667+0000","last_became_active":"2021-08-21T15:01:37.892137+0000","last_became_peered":"2021-08-21T15:01:37.892137+0000","last_unstale":"2021-08-21T15:01:51.885048+0000","last_undegraded":"2021-08-21T15:01:37.884863+0000","last_fullsized":"2021-08-21T15:01:37.775949+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-08-21T15:00:09.483517+0000","last_clean_scrub_stamp":"2021-08-21T15:00:09.483517+0000","log_size":4021,"ondisk_log_size":4021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40613,"num_objects":4021,"num_object_clones":0,"num_object_copies":8042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4021,"num_objects_misplaced":4021,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":4053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":205500,"num_objects":31665,"num_object_clones":0,"num_object_copies":63330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27742,"num_objects_misplaced":30306,"num_objects_unfound":0,"num_objects_dirty":31665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31665,"num_write_kb":31817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2641,"num_bytes_recovered":2641,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":281120768,"data_stored":416265,"data_compressed":149644,"data_compressed_allocated":155648,"data_compressed_original":311296,"omap_allocated":0,"internal_metadata":0},"log_size":31665,"ondisk_log_size":31665,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":115964117041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":98784247860,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":10,"seq":42949673034,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43220,"kb_used_data":10764,"kb_used_omap":0,"kb_used_meta":32448,"kb_avail":94328620,"statfs":{"total":96636764160,"available":96592506880,"internally_reserved":0,"allocated":11022336,"data_stored":76276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33226752},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 21 15:05:15 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.51200000000000001},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.63400000000000001,"5min":0.753,"15min":0.753},"last":0.441},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.48399999999999999,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.33500000000000002}]},{"osd":2,"last update":"Sat Aug 21 15:04:47 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45900000000000002,"15min":0.443},"min":{"1min":0.33600000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66300000000000003,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.46300000000000002,"15min":0.45100000000000001},"min":{"1min":0.28799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.311}]},{"osd":3,"last update":"Sat Aug 21 15:04:58 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.39300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.70599999999999996,"5min":0.752,"15min":0.752},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.5,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67200000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.50800000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705739,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43256,"kb_used_data":10736,"kb_used_omap":0,"kb_used_meta":32512,"kb_avail":94328584,"statfs":{"total":96636764160,"available":96592470016,"internally_reserved":0,"allocated":10993664,"data_stored":76587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33292288},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 21 15:05:18 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.45100000000000001,"15min":0.48499999999999999},"min":{"1min":0.32200000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.47199999999999998,"15min":0.56399999999999995},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.69099999999999995,"5min":1.821,"15min":1.821},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Aug 21 15:05:45 2021","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.30399999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.496,"15min":0.498},"min":{"1min":0.29899999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.66600000000000004,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.41599999999999998}]},{"osd":3,"last update":"Sat Aug 21 15:04:56 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.65700000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55600000000000005,"15min":0.57099999999999995},"min":{"1min":0.32000000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71999999999999997,"5min":0.88,"15min":0.88},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10760192,"data_stored":2627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10805248,"data_stored":2638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":129777664,"data_stored":205500,"data_compressed":74822,"data_compressed_allocated":77824,"data_compressed_original":155648,"omap_allocated":0,"internal_metadata":0}]}} 2021-08-21T15:05:54.411 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-08-21T15:05:54.414 DEBUG:teuthology.task.console_log:Killing console logger for smithi165 2021-08-21T15:05:54.415 DEBUG:teuthology.exit:Finished running handlers